A semantically guided video-audio generation method and system

Abstract

This invention provides a video-audio generation method and system based on semantic information guidance. The method includes: inputting video frames and video tag information (used as guidance for detecting the presence of sound objects) into a pre-trained language-image contrastive learning model to obtain the video hidden layer representation of the video frames and the text hidden layer representation of the tag information; determining the semantic similarity between the text hidden layer representation and the video hidden layer representation; using the semantic similarity as guidance to predict the timestamp of the sound generation node in the video; linearly projecting the video hidden layer representation into an audio synthesis model, and using a spatiotemporal information regulator to control the audio synthesis model to generate audio aligned with the video according to the timestamp. This invention uses video frames and tag information, and the similarity between video information and data tag information, to determine whether audio should be generated at each moment of the video, thereby generating audio with better granular alignment between video and audio by introducing guidance from data tag information.

Description

Technical Field

[0001] This invention relates to the field of intelligent voice, and in particular to a video-audio generation method and system based on semantic information guidance. Background Technology

[0002] Advances in V2A (Video-to-Audio) generation models have facilitated the development of AI-generated content, particularly in film and video post-processing applications. V2A generation has two key objectives: 1. Semantic alignment: the generated audio events should be consistent with the video content; 2. Temporal alignment: the generated audio should be synchronized with video frames. Existing V2A work attempts to improve generation performance in two directions. One direction is to employ more advanced networks, such as generative adversarial networks or autoregressive models. The other direction is to improve generation quality and controllability by combining various conditions. Specific applications include the Diff-Foley and Foley-Crafter models, which, given a video as input, generate audio related to the video content based on the video's information.

[0003] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0004] While existing methods demonstrate better temporal alignment in V2A generation, they still have certain limitations. Specifically, the aforementioned existing models cannot generate continuous, long-duration sounds for videos with moving visual presence (such as airplanes in flight and off-screen sirens). This indicates one aspect of the inadequacy of guidance: insufficient accuracy of temporal conditions. Another aspect stems from the low temporal resolution of semantic video conditions. For example, the resolution of video features in Diff-Foley is 4fps, far lower than the 30fps in ordinary videos. The low temporal resolution of video features leads to coarse synchronization between audio and video, affecting temporal alignment performance. Summary of the Invention

[0005] In order to at least address the problem that existing V2A models cannot generate continuous, long-duration audio for videos with moving visuals and have poor alignment.

[0006] In a first aspect, embodiments of the present invention provide a video-audio generation method guided by semantic information, comprising:

[0007] The video frames and the label information of the video used as guidance for detecting the appearance of sound objects are input into the contrastive learning language-image pre-trained model to obtain the video hidden layer representation of the video frames and the text hidden layer representation of the label information.

[0008] Determine the semantic similarity between the text hidden layer representation and the video hidden layer representation;

[0009] The semantic similarity is used as a guide to predict the timestamp of the sound utterance node in the video;

[0010] The hidden layer representation of the video is linearly projected into the audio synthesis model, and the spatiotemporal information adjuster is used to control the audio synthesis model to generate audio that is aligned with the video according to the timestamp.

[0011] Secondly, embodiments of the present invention provide a training method for a video-audio generation model guided by semantic information, comprising:

[0012] The video frames and the video's tag information are input into the video-audio generation model, wherein the video-audio generation model includes: a contrastive learning language-image pre-trained model, a visual encoder and frame adapter for applying the contrastive learning language-image pre-trained model to video-to-audio conversion, a frame adapter and a spatiotemporal information conditioner trained independently, and an audio synthesis model.

[0013] The encoder of the pre-trained language-image contrastive learning model is used to obtain the video hidden layer representation of the video frame and the text hidden layer representation of the label information.

[0014] The text latent layer representation and the video latent layer representation are adapted using a visual encoder with a projection layer and a frame adapter to determine the semantic similarity between the text latent layer representation and the video latent layer representation, wherein the projection layer is learnable and trainable.

[0015] The semantic similarity is used as a guide to predict the timestamp of the sound utterance node in the video;

[0016] The video hidden layer representation is linearly projected into the audio synthesis model using the frame adapter, and the spatiotemporal information regulator is used to control the audio synthesis model to generate predicted audio aligned with the video according to the timestamp;

[0017] The frame adapter and the spatiotemporal information conditioner are trained independently based on the diffusion loss determined by the predicted audio until the training requirements are met.

[0018] Thirdly, embodiments of the present invention provide a video-audio generation system guided by semantic information, comprising:

[0019] An embedding determination module is used to input video frames of a video and label information of the video used as guidance for detecting the presence of sound objects into a pre-trained model of contrastive learning language-image, so as to obtain the video hidden layer representation of the video frames and the text hidden layer representation of the label information;

[0020] A similarity determination module is used to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation;

[0021] The timestamp prediction module is used to predict the timestamp of the sound utterance node in the video, guided by the semantic similarity.

[0022] The generation module is used to linearly project the hidden layer representation of the video onto the audio synthesis model, and use a spatiotemporal information adjuster to control the audio synthesis model to generate audio aligned with the video according to the timestamp.

[0023] Fourthly, embodiments of the present invention provide a training system for a video-audio generation model guided by semantic information, comprising:

[0024] The input module is used to input video frames and the video's tag information into the video-audio generation model, wherein the video-audio generation model includes: a contrastive learning language-image pre-trained model, a visual encoder and frame adapter for applying the contrastive learning language-image pre-trained model to video-to-audio conversion, a frame adapter and a spatiotemporal information conditioner trained independently, and an audio synthesis model.

[0025] The embedding determination module is used to obtain the video hidden layer representation of the video frame and the text hidden layer representation of the label information through the encoder of the pre-trained language-image contrastive learning model;

[0026] A similarity determination module is used to adapt the text hidden layer representation and the video hidden layer representation by using a visual encoder with a projection layer and a frame adapter to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable.

[0027] The timestamp prediction module is used to predict the timestamp of the sound utterance node in the video, guided by the semantic similarity.

[0028] The generation module is used to linearly project the hidden layer representation of the video onto the audio synthesis model using the frame adapter, and to control the audio synthesis model to generate predicted audio aligned with the video according to the timestamp using a spatiotemporal information regulator.

[0029] The training module is used to independently train the frame adapter and the spatiotemporal information conditioner based on the diffusion loss determined by the predicted audio, until the training requirements are met.

[0030] Fifthly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the semantic information-guided video-audio generation and video-audio generation model training method according to any embodiment of the present invention.

[0031] In a sixth aspect, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the semantically information-guided video-audio generation method of any embodiment of the present invention.

[0032] In a seventh aspect, embodiments of the present invention provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the semantic information-guided video-audio generation and the training method of the video-audio generation model according to any embodiment of the present invention.

[0033] The beneficial effects of this invention are as follows: By utilizing existing data label information in the dataset and determining whether audio should be generated at each moment of the video based on the similarity between video information and data label information, audio with better granular alignment between video and audio is generated through the guidance of data label information. This enables video-audio synthesis in continuous sound scenarios, overcoming the limitation of existing video-audio generation models that only perform well in short-term events. It should be noted that the generation method of this invention relies on a semantically guided video-audio generation model to generate audio aligned with the video. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of a video-audio generation method based on semantic information guidance provided in an embodiment of the present invention;

[0036] Figure 2 This is a flowchart of a training method for a video-audio generation model based on semantic information guidance, provided in an embodiment of the present invention.

[0037] Figure 3This is a schematic diagram of the Smooth Foley model structure of a training method for a video-audio generation model based on semantic information guidance provided in an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of continuous label distribution for a training method of a video-audio generation model based on semantic information guidance provided in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the semantic alignment result of a training method for a video-audio generation model based on semantic information guidance provided in an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of the FAD results of different embedding extractors for a training method of a video-audio generation model based on semantic information guidance provided in an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram illustrating the subjective evaluation results of different models of a training method for a video-audio generation model based on semantic information guidance, provided in an embodiment of the present invention.

[0042] Figure 8 This is a schematic diagram comparing the time alignment of different models of a training method for a video-audio generation model based on semantic information guidance provided in an embodiment of the present invention.

[0043] Figure 9 This is a schematic diagram of the structure of a video-audio generation system based on semantic information guidance according to an embodiment of the present invention;

[0044] Figure 10 This is a schematic diagram of the structure of a training system for a video-audio generation model guided by semantic information, according to an embodiment of the present invention.

[0045] Figure 11 This is a schematic diagram of an embodiment of an electronic device for video-audio generation based on semantic information guidance, provided by an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] like Figure 1The diagram shows a flowchart of a video-audio generation method based on semantic information guidance according to an embodiment of the present invention, which includes the following steps:

[0048] S11: Input the video frames of the video and the label information of the video used as guidance for detecting the appearance of sound objects into the pre-trained model of contrastive learning language-image to obtain the video hidden layer representation of the video frames and the text hidden layer representation of the label information;

[0049] S12: Determine the semantic similarity between the text hidden layer representation and the video hidden layer representation;

[0050] S13: Use the semantic similarity as a guide to predict the timestamp of the sound-emitting node in the video;

[0051] S14: Linearly project the hidden layer representation of the video onto the audio synthesis model, and use the spatiotemporal information adjuster to control the audio synthesis model to generate audio aligned with the video according to the timestamp.

[0052] In this embodiment, the method considers using semantics to guide smooth and continuous V2A generation. Simply put, by using existing data label information in the dataset, the similarity between video information and data label information is used to determine whether audio should be generated at each moment of the video. This approach, guided by the introduction of data label information, generates audio with better video-audio granular alignment.

[0053] For step S11, in the video-to-audio generation process, the video to be used to generate audio, and the label information in the video used as guidance for detecting the presence of sound objects, are prepared. Specifically, the label information used as guidance for detecting the presence of sound objects includes multiple label information for different sound objects in the video. For example, in a video of a train, the label information could include train whistle, friction on rails, or other sound-producing content in the video. It is important to note that in this method, the input to the pre-trained language-image contrastive learning model must be video frames, not video segments. Specifically, this method uses frame-by-frame video guidance, rather than segment-by-segment guidance, to enhance the granularity of visual conditions. This is because frames inherently carry temporal information, thus enhancing temporal and semantic alignment; simultaneously, text label information serves as guidance for more accurate and coherent temporal conditions. Since a video may contain multiple objects, labels can serve as important guidance for detecting the presence of sound objects, making predictions more accurate than predictions from visual frames alone. The pre-trained model for contrastive learning of language and images can be CLIP (Contrastive Language-Image Pre-Training). The CLIP model learns how to compare text and images, thus achieving cross-modal understanding. This contrastive learning approach allows CLIP to learn the semantic relationships between text and images without any supervised labels. Then, through CLIP's encoder, hidden representations of the video and text (also known as video and text embeddings) are obtained.

[0054] For step S12, the semantic similarity between the text hidden layer representation and the video hidden layer representation is determined. Specifically, the semantic similarity is determined by the embedding cosine similarity between the text hidden layer representation and the video hidden layer representation, that is, the cosine similarity between the video frame embedding and the text embedding is determined, and the obtained cosine similarity is determined as the semantic similarity.

[0055] For step S13, specifically, the timestamps of the sound-producing nodes in the video are obtained by binarizing the similarity using a threshold of 0.5. For example, if the semantic similarity of a frame is greater than 0.5, it means that there should be sound at that timestamp. Through this continuous comparison, the timestamps of the sound-producing nodes in the entire video are obtained.

[0056] For step S14, the hidden layer representation of the video is linearly projected into the audio synthesis model as the basic material of the audio synthesis model. At the same time, the spatiotemporal information adjuster is used to control the audio synthesis model to obtain audio with high alignment with the video.

[0057] This implementation demonstrates that by leveraging existing data label information in the dataset and determining the similarity between video information and data label information to identify whether audio should be generated at each moment of the video, the introduction of data label information as a guide generates audio with better granular alignment between the video and audio. This enables video-audio synthesis in continuous sound scenarios, overcoming the limitation of existing video-audio generation models that only perform well in short-term events. It should be noted that this method relies on a semantically guided video-audio generation model to generate audio aligned with the video.

[0058] like Figure 2 The diagram shows a flowchart of a training method for a semantically guided video-audio generation model according to an embodiment of the present invention, including the following steps:

[0059] S21: Input the video frames and the video tag information into the video-audio generation model, wherein the video-audio generation model includes: a contrastive learning language-image pre-trained model, a visual encoder and frame adapter for applying the contrastive learning language-image pre-trained model to video-to-audio conversion, a frame adapter and a spatiotemporal information conditioner trained independently, and an audio synthesis model.

[0060] S22: The encoder of the pre-trained language-image contrastive learning model is used to obtain the video hidden layer representation of the video frame and the text hidden layer representation of the label information;

[0061] S23: Adapt the text hidden layer representation and the video hidden layer representation using a visual encoder with a projection layer and a frame adapter to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable;

[0062] S24: Using the semantic similarity as a guide, predict the timestamp of the sound-emitting node in the video;

[0063] S25: The hidden layer representation of the video is linearly projected into the audio synthesis model using the frame adapter, and the spatiotemporal information adjuster is used to control the audio synthesis model to generate predicted audio aligned with the video according to the timestamp;

[0064] S26: Based on the diffusion loss determined by the predicted audio, train the frame adapter and the spatiotemporal information conditioner independently until the training requirements are met.

[0065] In this embodiment, the method constructs a Smooth-Foley model, specifically called Creating Continuous Sound for Video-to-Audio Generation Under Semantic Guidance, which is a video-to-audio generation model guided by semantic information.

[0066] For step S21, Smooth Foley integrates a pre-trained T2A (text-to-audio) model through a lightweight adapter. For example, the Auffusion model can be chosen as the T2A model, or other models can be selected to adapt to situations with scarce data, while maintaining high fidelity and diverse audio synthesis capabilities. Figure 3 The diagram shows the structure of the Smooth-Foley model used in this method. The conditions input to the generative model come from two modules: a frame adapter and a spatiotemporal information conditioner. These two adapters are trained separately. While training one adapter, all other modules remain frozen. Video labels are merged to improve the accuracy of the temporal conditions. In the Smooth-Foley model of this method, semantic guidance originates from two aspects:

[0067] 1. This method uses frame-by-frame video guidance, rather than segment-by-segment guidance, to enhance the granularity of visual conditions. Since frames inherently carry temporal information, this process enhances temporal and semantic alignment;

[0068] 2. Text labels are used for more accurate and coherent temporal conditions. Since videos may contain multiple objects, labels can serve as important guidance for detecting the presence of sound objects, leading to more accurate predictions than those based solely on visual frames. This method's Smooth Foley also includes a frame adapter with frame-guided visual input, specifically comprising a visual encoder and a frame adapter.

[0069] For step S22, although CLIP has shown effectiveness in extracting visual semantic features in the visual encoder, this method needs to apply it to V2A (video-to-audio) generation. Therefore, this method uses an adapter to project frame features from CLIP, in the following format:

[0070]

[0071] Where v is the input video frame, The CLIP represents the frozen image encoder, and MLP represents the learnable projection module. This method follows the settings of the IP Adapter (a text-compatible image cue adapter for text-to-image diffusion models), using linear projection and embedding frames into the frozen T2A model.

[0072] In the frame adapter, this method integrates visual and textual features with frozen T2A through a parallel cross-attention adapter. Instead of using fragmented video embeddings as visual features, this method inputs the embeddings of the entire frame into the model. Weights are used... Combine these two outputs. Note that parallel crossovers can be formatted as follows:

[0073]

[0074]

[0075]

[0076] in, and These represent the extracted text embeddings (text hidden layer representation) and video frame embeddings (video hidden layer representation), respectively. During training, only... and It is trainable and enables lightweight adaptation, mapping pre-trained features to the latent space of the T2A model input. and It is initialized from a pre-trained cross-attention projection layer in Auffusion and kept frozen. The adapter is trained with a diffusion objective:

[0077]

[0078] For steps S23, S24, and S25, to improve temporal alignment, this method incorporates text label guidance into the temporal condition extraction. The CLIP features of each frame are mapped by a learnable projection layer, and the cosine similarity between the projected frame embedding and the text CLIP embedding is calculated. The timestamp representing the predicted vocal node is obtained by binarizing the similarity with a threshold of 0.5. The timestamps enhanced by label guidance can generate more time-synchronized predicted audio; the specific steps are similar to steps S12-S14 and will not be repeated here.

[0079] For step S26, the spatiotemporal information conditioner shares the same architecture as Auffusion's UNet encoder. During training, the input is a timestamp representing the predicted vocal node, processed based on semantic similarity, while the target is the corresponding audio. Similar to the frame adapter, the spatiotemporal information conditioner is trained using a diffusion loss. During inference, a temporal condition obtained through CLIP similarity is used to guide audio generation. For example, the diffusion loss is determined using pre-prepared baseline audio and predicted audio, and then the frame adapter and spatiotemporal information conditioner are trained independently using this diffusion loss. For instance, training may be required if the loss is less than a preset threshold; training ends when the loss is less than the preset threshold. Similarly, other loss training methods can also be used.

[0080] In terms of specific experimental preparation, our method found that existing models perform poorly on videos with continuous sound. To improve the generation performance of these video data, a subset with continuous sound was filtered from the V2A dataset VGGSound, termed VGGSound continuous. First, our method selects video clips with labels indicating continuous sound (e.g., sirens and airplane sounds). Then, text-to-audio grounding is used to filter out video clips whose audio does not match the labels. Ninety-five clips were manually selected as test segments. Label distribution and statistics are as follows: Figure 4 As shown, Smooth Foley is initialized from FoleyCrafter, and the adapter on VGGSound Continuous is fine-tuned to perform efficient training.

[0081] Regarding the baseline of the experiment, this method compares Smooth Foley with two state-of-the-art methods, Diff-Foley (DF) and FoleyCrafter (FC). Diff-Foley utilizes a CAVP (contrastive visual-audio pre-trained) encoder trained on video-audio pairs to synchronize V2A synthesis. FoleyCrafter is most similar to Smooth Foley, except that it uses segment-by-segment video embeddings instead of frame-by-frame video embeddings and does not include labels for temporal condition extraction.

[0082] Regarding evaluation metrics: This method employs several objective metrics to evaluate the performance of V2A generation, including FAD (Frechect Audio Distance), MKL (Mean KL Divergence), and CLIP score. FAD models the audio embedding as a Gaussian distribution and calculates the distance between the generated distribution and the ground truth distribution. FAD is calculated based on PANNs (large-scale pre-trained audio neural networks for audio pattern recognition), VGGish (an audio classification model implemented with Keras and TensorFlow backends), and CLIP. MKL measures the similarity at the paired sample level by calculating the average KL divergence across all classes in the test set. CLIP score uses Wav2CLIP to compare the similarity between the input video and the generated audio. Subjective evaluation is also performed, with 10 experienced human evaluators familiar with audio generation tasks scoring 10 samples for each model: 1. Semantic alignment; 2. Temporal alignment; 3. Audio quality. Scores range from 1 to 10.

[0083] Regarding the dataset, as mentioned above, the Smooth Foley algorithm used in this method was trained on VGGSound Continuous. This paper presents evaluation results on the entire VGGSound and VGGSound Continuous test sets, and compares them with existing techniques.

[0084] Comparison with existing technologies:

[0085] Quantitative comparison: Quantitative comparisons between different models are presented in the form of objective and subjective indicators. For example... Figure 5 (The semantic alignment results of the VGGSOUND-CONTINUOUS (VGG-C) and VGGSOUND (VGG) test splits, FC=FOLEYCRAFTER, DF=DIFF-FOLEY.) Figure 6 As shown, Smooth Foley achieves advanced semantic alignment with video content on both VGG and VGG-C, and provides better audio fidelity. In most cases, merging frame-by-frame video functions performs better than individual clip-by-clip video functions. Figure 7 The subjective evaluation results validated Smooth Foley's advantages in semantic alignment and audio quality, while further demonstrating better temporal alignment between the generated audio and the video input.

[0086] Qualitative comparison: in such Figure 8In this example, the timestamp comparison results of the sound-generating nodes for different models are presented. In the first example, FoleyCrafter failed to generate the engine sound when the plane approached. Diff-Foley produced a completely incorrect sound, but Smooth Foley successfully generated the corresponding sound effect when the object became blurred on the screen. Furthermore, the simulated audio of this method follows the Doppler effect, indicating that the frequency rises as the object approaches, peaks at the moment of closest encounter, and then declines. In contrast, FoleyCrafter exhibits the completely opposite pattern. In the second example, multiple events occurred. Steam whistled past after the train wheels screamed. SmoothFoley of this method accurately produced the steam whistling sound. The third example is a nearly static video accompanied by a siren. FoleyCrafter generated almost silent audio and failed to capture any semantic information from the visual frame. Smooth Foley successfully produced a high-quality and continuous siren sound. Diff Foley failed to generate sound correctly in all examples.

[0087] Temporal Condition Comparison: This method selected the first three and last three estimated probabilities from FoleyCrafter and Smooth Foley temporal detectors. In consecutive video frames, Smooth Foley consistently follows visual semantics (in the first case, a continuously moving train). When the main object (in the second case, a flying airplane) gradually becomes visually blurred, semantic guidance still captures visual cues and generates the correct temporal conditions.

[0088] In summary, this method proposes Smooth Foley in two ways, leveraging semantic guidance to improve the generation quality of continuous sound. The first approach is to replace segment-by-segment visual embedding with frame-by-frame visual embedding, improving the temporal resolution of visual guidance. The second approach is to incorporate text labels to predict more accurate temporal guidance. This method trains a frame adapter and a spatiotemporal information conditioner, respectively accepting semantic and temporal conditions, to effectively tune the pre-trained T2A model for V2A generation. This method also filters out VGGSound Continuous, focusing on videos with sound object blurring and sound sustainability. Based on VGGSound Continuous, the pre-trained V2A model for continuous sound generation is effectively enhanced. Experiments on VGGSound Continuous and VGGSound demonstrate that Smooth Foley outperforms baseline models in terms of audio quality, semantics, and temporal alignment.

[0089] like Figure 9The diagram shows a schematic of a semantic information-guided video-audio generation system according to an embodiment of the present invention. The system can execute the semantic information-guided video-audio generation method described in any of the above embodiments and is configured in a terminal.

[0090] This embodiment provides a video-audio generation system 10 based on semantic information guidance, which includes: an embedding determination module 11, a similarity determination module 12, a timestamp prediction module 13, and a generation module 14.

[0091] The embedding determination module 11 is used to input the video frames of the video and the label information of the video used as guidance for detecting the occurrence of sound objects into the pre-trained model of contrastive learning language-image to obtain the video hidden layer representation of the video frames and the text hidden layer representation of the label information; the similarity determination module 12 is used to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation; the timestamp prediction module 13 is used to use the semantic similarity as guidance to predict the timestamp of the sound node of the video; the generation module 14 is used to linearly project the video hidden layer representation into the audio synthesis model, and use the spatiotemporal information regulator to control the audio synthesis model to generate audio aligned with the video according to the timestamp.

[0092] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the semantic information-guided video-audio generation method in any of the above method embodiments.

[0093] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0094] The video frames and the label information of the video used as guidance for detecting the appearance of sound objects are input into the contrastive learning language-image pre-trained model to obtain the video hidden layer representation of the video frames and the text hidden layer representation of the label information.

[0095] Determine the semantic similarity between the text hidden layer representation and the video hidden layer representation;

[0096] The semantic similarity is used as a guide to predict the timestamp of the sound utterance node in the video;

[0097] The hidden layer representation of the video is linearly projected into the audio synthesis model, and the spatiotemporal information adjuster is used to control the audio synthesis model to generate audio that is aligned with the video according to the timestamp.

[0098] like Figure 10The diagram shows a structural schematic of a training system for a video-audio generation model guided by semantic information according to an embodiment of the present invention. The system can execute the training method for the video-audio generation model guided by semantic information as described in any of the above embodiments and is configured in a terminal.

[0099] This embodiment provides a training system 20 for a video-audio generation model guided by semantic information, which includes: an input module 21, an embedding determination module 22, a similarity determination module 23, a timestamp prediction module 24, a generation module 25, and a training module 26.

[0100] The input module 21 is used to input video frames and the video's tag information into the video-audio generation model. The video-audio generation model includes: a contrastive learning language-image pre-trained model, a visual encoder and frame adapter for applying the contrastive learning language-image pre-trained model to video-to-audio conversion, independently trained frame adapters and spatiotemporal information conditioners, and an audio synthesis model. The embedding determination module 22 is used to obtain the video hidden layer representation of the video frames and the text hidden layer representation of the tag information using the encoder of the contrastive learning language-image pre-trained model. The similarity determination module 23 is used to perform similarity determination on the text using a visual encoder and frame adapter with a projection layer. The hidden layer representation and the video hidden layer representation adaptation process determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable; the timestamp prediction module 24 is used to use the semantic similarity as a guide to predict the timestamp of the sound node in the video; the generation module 25 is used to linearly project the video hidden layer representation into the audio synthesis model using the frame adapter, and use the spatiotemporal information regulator to control the audio synthesis model to generate predicted audio aligned with the video according to the timestamp; the training module 26 is used to independently train the frame adapter and the spatiotemporal information regulator based on the diffusion loss determined by the predicted audio until the training requirements are met.

[0101] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the training method for the semantically guided video-audio generation model in any of the above method embodiments.

[0102] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0103] The video frames and the video's tag information are input into the video-audio generation model, wherein the video-audio generation model includes: a contrastive learning language-image pre-trained model, a visual encoder and frame adapter for applying the contrastive learning language-image pre-trained model to video-to-audio conversion, a frame adapter and a spatiotemporal information conditioner trained independently, and an audio synthesis model.

[0104] The encoder of the pre-trained language-image contrastive learning model is used to obtain the video hidden layer representation of the video frame and the text hidden layer representation of the label information.

[0105] The text latent layer representation and the video latent layer representation are adapted using a visual encoder with a projection layer and a frame adapter to determine the semantic similarity between the text latent layer representation and the video latent layer representation, wherein the projection layer is learnable and trainable.

[0106] The semantic similarity is used as a guide to predict the timestamp of the sound utterance node in the video;

[0107] The video hidden layer representation is linearly projected into the audio synthesis model using the frame adapter, and the spatiotemporal information regulator is used to control the audio synthesis model to generate predicted audio aligned with the video according to the timestamp;

[0108] The frame adapter and the spatiotemporal information conditioner are trained independently based on the diffusion loss determined by the predicted audio until the training requirements are met.

[0109] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they execute the semantic information-guided video-audio generation and semantic information-guided video-audio generation model training methods in any of the above method embodiments.

[0110] Figure 11 This is a schematic diagram of the hardware structure of an electronic device using a semantically information-guided video-audio generation method according to another embodiment of this application, as shown below. Figure 11 As shown, the device includes:

[0111] One or more processors 1110 and memory 1120, Figure 11 Taking a processor 1110 as an example, the device for the semantic information-guided video-audio generation method may further include an input device 1130 and an output device 1140.

[0112] The processor 1110, memory 1120, input device 1130, and output device 1140 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0113] The memory 1120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the semantic information-guided video-audio generation method in the embodiments of this application. The processor 1110 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1120, thereby implementing the semantic information-guided video-audio generation method described in the above embodiments.

[0114] The memory 1120 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1120 may optionally include memory remotely located relative to the processor 1110, and these remote memories may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0115] Input device 1130 can receive input numerical or character information. Output device 1140 may include display devices such as a display screen.

[0116] The one or more modules are stored in the memory 1120, and when executed by the one or more processors 1110, they execute the semantic information-guided video-audio generation method in any of the above method embodiments.

[0117] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0118] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0119] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the semantic information-guided video-audio generation method of any embodiment of this invention.

[0120] The electronic devices described in this application exist in various forms, including but not limited to:

[0121] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0122] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0123] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0124] (4) Other electronic devices with data processing functions.

[0125] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video-audio generation method guided by semantic information, comprising: The video frames and the label information of the video used as guidance for detecting the appearance of sound objects are input into the pre-trained model of contrastive learning language-image to obtain the video hidden layer representation of the video frames and the text hidden layer representation of the label information. The label information used as guidance for detecting the appearance of sound objects includes multiple label information of different sound objects in the video. The semantic similarity between the text latent layer representation and the video latent layer representation is determined, wherein the semantic similarity is determined by the embedding cosine similarity between the text latent layer representation and the video latent layer representation; The semantic similarity is used as a guide to predict the timestamp of the sound utterance node in the video; The hidden layer representation of the video is linearly projected into the audio synthesis model, and the spatiotemporal information adjuster is used to control the audio synthesis model to generate audio that is aligned with the video according to the timestamp.

2. A training method for a video-audio generation model guided by semantic information, comprising: The video frames and the video's tag information are input into the video-audio generation model, wherein the video-audio generation model includes: a contrastive learning language-image pre-trained model, a visual encoder and frame adapter for applying the contrastive learning language-image pre-trained model to video-to-audio conversion, a frame adapter and a spatiotemporal information conditioner trained independently, and an audio synthesis model. The encoder of the pre-trained language-image contrastive learning model is used to obtain the video hidden layer representation of the video frame and the text hidden layer representation of the label information. The text latent layer representation and the video latent layer representation are adapted using a visual encoder with a projection layer and a frame adapter to determine the semantic similarity between the text latent layer representation and the video latent layer representation, wherein the projection layer is learnable and trainable. The semantic similarity is used as a guide to predict the timestamp of the sound utterance node in the video; The video hidden layer representation is linearly projected into the audio synthesis model using the frame adapter, and the spatiotemporal information regulator is used to control the audio synthesis model to generate predicted audio aligned with the video according to the timestamp; The frame adapter and the spatiotemporal information conditioner are trained independently based on the diffusion loss determined by the predicted audio until the training requirements are met.

3. The method according to claim 2, wherein, The projection layer is trained simultaneously when the frame adapter is trained, and the audio synthesis model is trained simultaneously when the spatiotemporal information conditioner is trained.

4. A video-audio generation system guided by semantic information, comprising: An embedding determination module is used to input video frames of a video and label information of the video used as guidance for detecting the presence of sound objects into a pre-trained model of contrastive learning language-image, so as to obtain the video hidden layer representation of the video frames and the text hidden layer representation of the label information, wherein the label information used as guidance for detecting the presence of sound objects includes: multiple label information of different sound objects in the video; A similarity determination module is used to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the semantic similarity is determined by the embedding cosine similarity between the text hidden layer representation and the video hidden layer representation; The timestamp prediction module is used to predict the timestamp of the sound utterance node in the video, guided by the semantic similarity. The generation module is used to linearly project the hidden layer representation of the video onto the audio synthesis model, and use a spatiotemporal information adjuster to control the audio synthesis model to generate audio aligned with the video according to the timestamp.

5. A training system for a video-audio generation model guided by semantic information, comprising: The input module is used to input video frames and the video's tag information into the video-audio generation model, wherein the video-audio generation model includes: a contrastive learning language-image pre-trained model, a visual encoder and frame adapter for applying the contrastive learning language-image pre-trained model to video-to-audio conversion, a frame adapter and a spatiotemporal information conditioner trained independently, and an audio synthesis model. An embedding determination module is used to obtain the video hidden layer representation of the video frame and the text hidden layer representation of the label information through the encoder of the pre-trained language-image contrastive learning model; A similarity determination module is used to adapt the text hidden layer representation and the video hidden layer representation by using a visual encoder with a projection layer and a frame adapter to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable. The timestamp prediction module is used to predict the timestamp of the sound utterance node in the video, guided by the semantic similarity. The generation module is used to linearly project the hidden layer representation of the video onto the audio synthesis model using the frame adapter, and to control the audio synthesis model to generate predicted audio aligned with the video according to the timestamp using a spatiotemporal information regulator. The training module is used to independently train the frame adapter and the spatiotemporal information conditioner based on the diffusion loss determined by the predicted audio, until the training requirements are met.

6. The system according to claim 5, wherein, The training module is used for: The projection layer is trained simultaneously when the frame adapter is trained, and the audio synthesis model is trained simultaneously when the spatiotemporal information conditioner is trained.

7. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-3.

8. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-3.

9. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-3.

Images