Automatic Audio Generation System for Web Comics Based on Visual Language Model

Abstract

An automatic audio generation system for webcomics based on a visual language model is disclosed. This system receives elongated webcomic images through a visual language analysis module and segments them into multiple images corresponding to sequence nodes according to image frames. The module extracts the semantic, emotional features, and events of each image and stores them as a feature event list. Subsequently, a background music generation module generates background music based on this list and labels the sequence nodes, while a sound effect generation module generates corresponding local event sound effects. Finally, an output module packages the images and audio into a webcomic file. When an electronic device displays a specific sequence of images, the corresponding background music and sound effects are automatically played, achieving an immersive audio-visual synchronized experience.

Claims

1. An automatic audio generation system for webcomics based on a visual language model, comprising: an electronic device storing a strip-shaped webcomic image; a server connected to the electronic device, the server comprising: a visual language analysis module, which segments the strip-shaped webcomic image into a plurality of sequence images, each sequence image corresponding to a sequence node; and stores them in order of the sequence nodes into a feature event list, the feature event list including a semantic feature, an emotional feature, and an image event; a background music generation module, which generates background music based on the feature event list; a sound effect generation module, which generates a plurality of local event sound effects based on the feature event list and assigns the local event sound effects to the corresponding sequence nodes; and an output module, which encapsulates the strip-shaped webcomic image, the local event sound effects, and the background music according to the sequence nodes, and outputs a webcomic file to the electronic device.

2. The webcomic audio automatic generation system based on visual language model as described in claim 1, wherein the server further includes a scrolling trigger module for setting the playback timing of the sequence images, the background music, and the local event sound effects.

3. As described in claim 2, in the webcomic audio automatic generation system based on visual language model, when one of the sequential images of the webcomic file is displayed on a screen of one of the electronic devices and the timing of the playback is met, the electronic device plays the background music and the local event sound effects corresponding to the sequence node of the sequence image.

4. The webcomic audio automatic generation system based on visual language model as described in claim 3, wherein the playback timing includes triggering the playback of the corresponding background music or local event sounds by entering a specific position on the display screen based on one of the image frames in the elongated webcomic image.

5. The webcomic audio automatic generation system based on visual language model as described in claim 1, wherein the server further includes a remastering module, wherein the background music generation module generates a plurality of alternative background music based on the feature event list, and receives a regeneration instruction via the remastering module to adjust a random parameter, thereby regenerating the audio content.

6. The webcomic audio automatic generation system based on visual language model as described in claim 1, wherein the visual language analysis module cuts the long strip webcomic image in the following ways: precise cutting based on a single image frame, cutting by combining multiple adjacent image frames, or cutting within a certain range based on image frames within a pre-set fixed length.

7. The webcomic audio automatic generation system based on visual language model as described in claim 3, wherein when the electronic device detects that the background music has been played to the end and the reading progress of one of the display screens is still in the sequence of images corresponding to the background music, the background music is played in a loop.

Images