Video-audio generation method and system based on semantic information guidance

Abstract

The embodiment of the invention provides a video-audio generation method and system based on semantic information guidance. The method comprises the following steps: inputting a video frame and label information of a video, which is used for detecting sound object appearance guidance, into a pre-training model of a contrast learning language-image to obtain a video hidden layer representation of the video frame and a text hidden layer representation of the label information; determining the semantic similarity of the text hidden layer representation and the video hidden layer representation; taking the semantic similarity as guidance to predict a timestamp of a sounding node of the video; and linearly projecting the video hidden layer representation into the audio synthesis model, and controlling the audio synthesis model to generate an audio aligned with the video according to the timestamp by using a spatio-temporal information regulator. According to the embodiment of the invention, whether the audio needs to be generated at each moment of the video is determined according to the video frame, the label information and the similarity between the video information and the data label information, so that the audio with better video-audio granularity alignment is generated by introducing the guidance of the data label information.

Description

Technical Field

[0001] The present invention relates to the field of intelligent speech, and in particular to a method and system for generating video and audio based on semantic information guidance. Background Art

[0002] Advances in V2A (Video-to-Audio) generation models have promoted the development of AI-generated content, especially in applications such as film and video post-processing. V2A generation has two key goals: 1. Semantic alignment: the generated sound events should be consistent with the video content; 2. Temporal alignment: the generated sound should be synchronized with the video frames. Existing V2A work attempts to improve power generation performance from two directions. One direction is to adopt more advanced networks, such as generative adversarial networks or autoregressive models. The other direction is to improve the quality and controllability of power generation by combining various conditions. Specific applications include Diff-Foley and Foley-Crafter models, which generate audio related to the video content based on the information of the video by inputting the video.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art: Although existing methods show better temporal alignment in V2A generation, they still have certain limitations. Specifically, the above existing models are unable to generate continuous, long-duration sounds for videos with moving visual presence, such as flying airplanes and sound sirens off-screen. This indicates one aspect of insufficient guidance: the temporal conditioning is not accurate enough. Another aspect stems from the low temporal resolution of semantic video conditioning. For example, the resolution of video features in Diff-Foley is 4fps, which is much lower than the 30fps in normal videos. The low temporal resolution of video features leads to rough synchronization between audio and video, affecting the temporal alignment performance. Summary of the invention

[0004] In order to at least solve the problem that existing V2A models in the prior art cannot generate continuous and long-term sounds for videos with moving visual presence and have poor alignment.

[0005] In a first aspect, an embodiment of the present invention provides a video-audio generation method based on semantic information guidance, comprising: Inputting a video frame of a video and label information of the video used as a guide for detecting the presence of a sound object into a pre-trained model for contrastive learning of language-image, thereby obtaining a video latent layer representation of the video frame and a textual latent layer representation of the label information; Determining the semantic similarity between the text latent representation and the video latent representation; Using the semantic similarity as a guide, predicting the timestamp of the sounding node of the video; The video hidden layer representation is linearly projected into an audio synthesis model, and a spatiotemporal information regulator is used to control the audio synthesis model according to the timestamp to generate audio aligned with the video.

[0006] In a second aspect, an embodiment of the present invention provides a method for training a video-audio generation model guided by semantic information, comprising: Inputting video frames of a video and label information of the video into the video-audio generation model, wherein the video-audio generation model includes: a pre-trained model for contrastive learning of language-image, a visual encoder and a frame adapter for applying the pre-trained model for contrastive learning of language-image to video-audio conversion, a frame adapter and a spatiotemporal information regulator that are independently trained, and an audio synthesis model; Obtaining a video latent layer representation of the video frame and a text latent layer representation of the label information through an encoder of the pre-trained model of the comparative learning language-image; Adapting the text hidden layer representation and the video hidden layer representation using a visual encoder and a framework adapter with a projection layer to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable; Using the semantic similarity as a guide, predicting the timestamp of the sounding node of the video; Using the frame adapter to linearly project the video hidden layer representation into the audio synthesis model, and using the spatiotemporal information regulator to control the audio synthesis model according to the timestamp to generate predicted audio aligned with the video; The frame adapter and the spatiotemporal information regulator are trained independently based on the diffusion loss determined by the predicted audio until the training requirements are met.

[0007] In a third aspect, an embodiment of the present invention provides a video-audio generation system based on semantic information guidance, including: An embedding determination module is used to input a video frame of a video and label information of the video used as a guide for detecting the presence of a sound object into a pre-trained model for contrastive learning of language-image, so as to obtain a video latent layer representation of the video frame and a textual latent layer representation of the label information; A similarity determination module, used to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation; A timestamp prediction module, used to predict the timestamp of the sounding node of the video using the semantic similarity as a guide; A generation module is used to linearly project the video hidden layer representation into the audio synthesis model, and use the spatiotemporal information regulator to control the audio synthesis model according to the timestamp to generate audio aligned with the video.

[0008] In a fourth aspect, an embodiment of the present invention provides a training system for a video-audio generation model guided by semantic information, comprising: An input module, used for inputting video frames of a video and label information of the video into the video-audio generation model, wherein the video-audio generation model includes: a pre-trained model for contrastive learning of language-image, a visual encoder and a frame adapter for applying the pre-trained model for contrastive learning of language-image to video-audio conversion, a frame adapter and a spatiotemporal information regulator that are independently trained, and an audio synthesis model; An embedding determination module, configured to obtain a video latent layer representation of the video frame and a text latent layer representation of the label information through an encoder of a pre-trained model of language-image through comparative learning; A similarity determination module is used to adapt the text hidden layer representation and the video hidden layer representation using a visual encoder with a projection layer and a framework adapter to determine the semantic similarity of the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable; A timestamp prediction module, used to predict the timestamp of the sounding node of the video using the semantic similarity as a guide; A generation module, configured to linearly project the video hidden layer representation into an audio synthesis model using the frame adapter, and to control the audio synthesis model to generate predicted audio aligned with the video according to the timestamp using a spatiotemporal information regulator; A training module is used to train the frame adapter and the spatiotemporal information regulator independently based on the diffusion loss determined by the predicted audio until the training requirements are met.

[0009] In a fifth aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method for training a video-audio generation and a video-audio generation model based on semantic information guidance according to any embodiment of the present invention.

[0010] In a sixth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the video-audio generation method based on semantic information guidance of any embodiment of the present invention are implemented.

[0011] In the seventh aspect, an embodiment of the present invention provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the method for training a video-audio generation and a video-audio generation model based on semantic information guidance of any embodiment of the present invention are implemented.

[0012] The beneficial effect of the embodiment of the present invention is that: through the existing data label information in the data set, the similarity between the video information and the data label information is used to confirm whether audio should be generated at each moment of the video, thereby generating audio with better video-audio granularity alignment by introducing the guidance of the data label information, thereby achieving video-audio synthesis in a continuous sound scene, making up for the deficiency that the video-audio generation model in the prior art can only have a good effect in short-term events. It should be noted that the generation method of this method relies on the video-audio generation model based on semantic information guidance constructed by this method to generate audio aligned with the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 is a flow chart of a method for generating video-audio based on semantic information guidance provided by one embodiment of the present invention; Figure 2 It is a flowchart of a method for training a video-audio generation model based on semantic information guidance provided by one embodiment of the present invention; Figure 3 It is a schematic diagram of the Smooth Foley model structure of a training method of a video-audio generation model guided by semantic information provided by one embodiment of the present invention; Figure 4 It is a schematic diagram of continuous label distribution of a training method of a video-audio generation model guided by semantic information provided by an embodiment of the present invention; Figure 5 It is a schematic diagram of semantic alignment results of a training method for a video-audio generation model guided by semantic information provided by one embodiment of the present invention; Figure 6 It is a schematic diagram of FAD results of different embedding extractors of a training method of a video-audio generation model guided by semantic information provided by an embodiment of the present invention; Figure 7It is a schematic diagram of subjective evaluation results of different models of a training method for a video-audio generation model guided by semantic information provided by an embodiment of the present invention; Figure 8 It is a comparative schematic diagram of time alignment of different models in a training method of a video-audio generation model guided by semantic information provided by an embodiment of the present invention; Fig. 9 It is a schematic diagram of the structure of a video-audio generation system based on semantic information guidance provided by an embodiment of the present invention; Fig.10 It is a structural diagram of a training system for a video-audio generation model guided by semantic information provided by an embodiment of the present invention; Fig.11 A schematic structural diagram of an embodiment of an electronic device for video-audio generation guided by semantic information provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] like Figure 1 FIG. 1 is a flowchart of a method for generating video-audio based on semantic information guidance provided by an embodiment of the present invention, comprising the following steps: S11: Inputting a video frame of a video and label information of the video used as a guide for detecting the appearance of a sound object into a pre-trained model of contrastive learning language-image, and obtaining a video latent layer representation of the video frame and a textual latent layer representation of the label information; S12: Determine the semantic similarity between the text hidden layer representation and the video hidden layer representation; S13: using the semantic similarity as a guide to predict the timestamp of the sounding node of the video; S14: Linearly projecting the video hidden layer representation into the audio synthesis model, and using the spatiotemporal information regulator to control the audio synthesis model according to the timestamp to generate audio aligned with the video.

[0017] In this embodiment, the method considers using semantics to guide smooth and continuous V2A generation. In short, through the existing data label information in the data set, the similarity between the video information and the data label information is used to confirm whether audio should be generated at each moment of the video, thereby generating audio with better video-audio granularity alignment by introducing the guidance of the data label information.

[0018] For step S11, in the process of generating video to audio, prepare the video to generate audio and the label information in the video used as a guide for detecting the appearance of sound objects. Specifically, the label information used as a guide for detecting the appearance of sound objects includes: multiple label information of different sound objects in the video. For example, in the video of the train, the label information can include the train whistle, the friction rails, and other content that makes sounds in the video. It should be noted that in the pre-trained model of input contrast learning language-image, the input of this method must be video frames, not video clips. Specifically, this method uses frame-by-frame video guidance instead of segment-by-segment guidance to enhance the granularity of visual conditions. This is because frames inherently carry time information, so the process enhances time and semantic alignment; at the same time, the text label information serves as a guide for more accurate and coherent time conditions. Since the video may contain multiple objects, the label can serve as an important guide for detecting the appearance of sound objects, and such predictions are more accurate than predictions from visual frames alone. Among them, the pre-training model for contrastive learning language-image can be CLIP (Contrastive Language-Image Pre-Training). The CLIP model achieves cross-modal understanding by learning how to compare text and images. This contrastive learning method enables CLIP to learn the semantic relationship between text and image without any supervised labels. In this way, the latent layer representation of video and text (also called embedding of video and text) is obtained by encoding through the CLIP encoder.

[0019] For step S12, the semantic similarity of the text hidden layer representation and the video hidden layer representation is determined. Specifically, the semantic similarity is determined by the embedding cosine similarity of the text hidden layer representation and the video hidden layer representation, that is, the cosine similarity of the video frame embedding and the text embedding is determined, and the obtained cosine similarity is determined as the semantic similarity.

[0020] For step S13, specifically, the timestamp of the sound node of the video is obtained by binarizing the similarity with a threshold of 0.5. For example, when the semantic similarity of a certain frame is greater than 0.5, it means that there should be sound at the timestamp. Through such continuous comparison, the timestamp of the sound node of the entire video is obtained.

[0021] For step S14, the video hidden layer representation is linearly projected into the audio synthesis model as the basic material of the audio synthesis model. At the same time, the audio synthesis model is controlled by the spatiotemporal information regulator to obtain audio that is highly aligned with the video.

[0022] It can be seen from this implementation that the existing data label information in the data set is used to confirm whether audio should be generated at each moment of the video by the similarity between the video information and the data label information, thereby generating audio with better video-audio granularity alignment by introducing the guidance of the data label information, thereby achieving video-audio synthesis in a continuous sound scene, making up for the deficiency that the video-audio generation model in the prior art can only have a good effect in short-term events. It should be noted that the generation method of this method relies on the video-audio generation model based on semantic information guidance constructed by this method to generate audio aligned with the video.

[0023] like Figure 2 FIG. 1 is a flowchart of a method for training a video-audio generation model based on semantic information guidance provided by an embodiment of the present invention, comprising the following steps: S21: inputting video frames of a video and label information of the video into the video-audio generation model, wherein the video-audio generation model includes: a pre-trained model for contrastive learning of language-image, a visual encoder and a frame adapter for applying the pre-trained model for contrastive learning of language-image to video-audio conversion, a frame adapter and a spatiotemporal information regulator that are independently trained, and an audio synthesis model; S22: Obtaining a video latent layer representation of the video frame and a text latent layer representation of the label information through an encoder of the pre-trained model of the comparative learning language-image; S23: using a visual encoder with a projection layer and a framework adapter to adapt the text hidden layer representation and the video hidden layer representation to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable; S24: using the semantic similarity as a guide to predict the timestamp of the sounding node of the video; S25: linearly projecting the video hidden layer representation into the audio synthesis model using the frame adapter, and controlling the audio synthesis model to generate predicted audio aligned with the video according to the timestamp using the spatiotemporal information regulator; S26: The frame adapter and the spatiotemporal information regulator are trained independently based on the diffusion loss determined by the predicted audio until the training requirements are met.

[0024] In this embodiment, the method constructs a Smooth-Foley model, wherein the Smooth-Foley model is specifically called Creating Continuous Sound for Video-to-Audio Generation Under Semantic Guidance, which is a video-to-audio generation model guided by semantic information.

[0025] For step S21, Smooth Foley integrates a pre-trained T2A (text-to-audio) model through a lightweight adapter. For example, the Auffusion model can be selected as the T2A model, or other models can be selected to adapt to data scarcity while maintaining high fidelity and diverse audio synthesis capabilities. Figure 3 The structure of the Smooth-Foley model of this method is shown. The conditions input to the generative model come from two modules: the frame adapter and the spatiotemporal information regulator. The two adapters are trained separately. When training one of the adapters, all other modules remain frozen. Video labels are incorporated to improve the accuracy of the temporal conditions. In the Smooth Foley of this method, semantic guidance comes from two aspects: 1. Our method uses frame-by-frame video guidance instead of segment-by-segment guidance to enhance the granularity of visual conditions. Since frames inherently carry temporal information, this process enhances temporal and semantic alignment; 2. Text labels are used for more accurate and coherent temporal conditions. Since videos may contain multiple objects, labels can serve as an important guide for detecting the appearance of sound objects, and such predictions are more accurate than predictions from visual frames alone. Smooth Foley of this method also includes a frame adapter with frame vision guidance, specifically including: a visual encoder and a frame adapter.

[0026] For step S22, in the visual encoder, although CLIP has shown effectiveness in extracting visual semantic features, this method needs to apply it to V2A (video-to-audio) generation. Therefore, this method uses an adapter to project frame features from CLIP, with the following format: Where v is the input video frame, represents the frozen CLIP image encoder and MLP represents the learnable projection module. This method follows the setting of IP Adapter (Text-Compatible Image Prompt Adapter for Text-to-Image Diffusion Model) and uses linear projection and embeds the frames into the frozen T2A model.

[0027] In the frame adapter, our method integrates visual features and textual features with frozen T2A through parallel criss-cross attention adapters. Instead of using episodic video embeddings as visual features, our method inputs the embeddings of the entire frame into the model. Using weights Combine these two outputs. Parallel cross-notes can be formatted as: in, and denote the extracted text embedding (text hidden layer representation) and video frame embedding (video hidden layer representation) respectively. During the training process, only and It is trainable and enables lightweight adaptation to map pre-trained features to the latent space of the T2A model input. and It is initialized from a pre-trained criss-cross attention projection layer from Auffusion and kept frozen. The adapter is trained with a diffusion objective: For steps S23, S24, and S25, in order to improve time alignment, the method incorporates the guidance of text labels into the temporal condition extraction. The CLIP features of each frame are mapped by a learnable projection layer, and the cosine similarity between the projected frame embedding and the text CLIP embedding is calculated. The timestamp representing the predicted utterance node is obtained by binarizing the similarity with a threshold of 0.5. The timestamp enhanced by label guidance can generate more time-synchronized predicted audio. The specific steps are similar to steps S12-S14 and will not be repeated here.

[0028] For step S26, the spatiotemporal information regulator shares the same architecture as Auffusion's UNet encoder. During the training process, the input is a timestamp representing a predicted utterance node after processing based on semantic similarity, and the target is the corresponding audio. Similar to the frame adapter, the spatiotemporal information regulator is trained by diffusion loss. During the inference process, the time conditions obtained by CLIP similarity are used to guide audio generation. For example, the diffusion loss is determined using pre-prepared reference audio and predicted audio, and then the diffusion loss is used to train the frame adapter and the spatiotemporal information regulator independently. For example, the training requirement is that the loss is less than a preset threshold, then when the loss is less than the preset threshold, the training ends. Similarly, other loss training methods can also be used.

[0029] Specific experimental preparation, this method found that the existing model performed poorly on videos with continuous sound. In order to improve the generation performance of these video data, a subset with continuous sound is filtered out from the V2A dataset VGGSound, called VGGSound continuous. First, this method selects video clips with labels indicating continuous sounds (for example, sirens and airplane sounds). Then, text-to-audio grounding is used to filter out video clips whose audio does not match the labels. 95 clips were manually selected as test splits. Label distribution and statistics are shown in Figure 2. Figure 4 Initialize Smooth Foley from FoleyCrafter and fine-tune the adapter on VGGSound Continuous to perform efficient training.

[0030] Regarding the experimental baseline, this method compares Smooth Foley with two state-of-the-art methods in the prior art, Diff-Foley (DF) and FoleyCrafter (FC). Diff-Foley uses a CAVP (contrastive visual-audio pre-trained) encoder to synchronize V2A synthesis. FoleyCrafter is most similar to Smooth Foley, except that it uses segment-by-segment video embedding instead of frame-by-frame video embedding and does not include labels for temporal conditional extraction.

[0031] About evaluation metrics: This method uses several objective metrics to evaluate the performance of V2A generation, including FAD (Frechect Audio Distance), MKL (Mean KL Divergence), and CLIP score. FAD models audio embedding as a Gaussian distribution and calculates the distance between the generated distribution and the ground truth distribution. FAD based on PANNs (large-scale pre-trained audio neural network for audio pattern recognition), VGGish (audio classification model implemented by Keras and TensorFlow backend), and CLAP were calculated respectively. MKL measures the similarity at the paired sample level by calculating the average KL divergence of all categories in the test set. CLIP score compares the similarity between the input video and the generated audio using Wav2CLIP. This method also conducted a subjective evaluation, inviting 10 experienced human evaluators familiar with audio generation tasks to score 10 samples of each model: 1. semantic alignment; 2. temporal alignment; 3. audio quality. The score ranges from 1 to 10.

[0032] Regarding the dataset, as mentioned above, our Smooth Foley is trained on VGGSound Continuous. We show the evaluation results on the entire VGGSound and VGGSound Continuous test sets and compare them with the prior art.

[0033] Comparison with existing technologies: Quantitative comparison: Quantitative comparisons between different models are presented in the form of objective and subjective indicators. Figure 5 (Semantic alignment results of VGGSOUND-CONTINUOUS (VGG-C) and VGGSOUND (VGG) test splits, FC=FOLEYCRAFTER, DF=DIFF-FOLEY.) Figure 6 As shown in , on both VGG and VGG-C, Smooth Foley achieves high-level semantic alignment with the video content and provides better audio fidelity. In most cases, merging frame-by-frame video features performs better than single clip-by-clip video features. Figure 7 The subjective evaluation results in validate the advantages of Smooth Foley in terms of semantic alignment and audio quality, while further showing better temporal alignment of the generated audio with the video input.

[0034] Qualitative comparison: Figure 8 In Figure 1, we present a comparison of the timestamps of the sound-generating nodes of different models. In the first example, FoleyCrafter fails to produce engine sounds as a plane approaches. Diff-Foley produces completely incorrect sounds, but Smooth Foley successfully generates the corresponding sound effects when the object becomes blurred on the screen. In addition, our simulated audio follows the principle of the Doppler effect, which shows that the frequency rises as the object approaches, peaks at the closest encounter moment, and then decreases. In contrast, FoleyCrafter presents the exact opposite pattern. In the second example, multiple events occur. After the train wheels scream, steam whistles past. Our SmoothFoley is able to accurately produce the steam whistle. The third example is a nearly static video accompanied by a police siren. FoleyCrafter generates almost silent audio, completely failing to capture semantic information from the visual frame. Smooth Foley successfully produces high-quality and continuous siren sounds. Diff Foley fails to generate sounds correctly in all examples.

[0035] Temporal Condition Comparison: Our method selects the top 3 and bottom 3 estimated probabilities from the temporal detectors FoleyCrafter and Smooth Foley. In consecutive video frames, Smooth Foley always follows the visual semantics (in the first case, a continuously moving train). When the main object (in the second case, a flying plane) gradually becomes visually blurred, the semantic guidance still captures the visual cues and generates the correct temporal conditions.

[0036] In summary, this method proposes Smooth Foley from two aspects to improve the generation quality of continuous sound using semantic guidance. The first approach is to replace segment-by-segment visual embeddings with frame-by-frame visual embeddings to improve the temporal resolution of visual guidance. The second is to combine text labels to predict more accurate temporal guidance. This method trains a frame adapter and a spatiotemporal information regulator, which accept semantic and temporal conditions respectively, to effectively adjust the pre-trained T2A model to generate V2A. This method also filters out VGGSound Continuous and focuses on videos with blurred sound objects and sound sustainability. Based on VGGSound Continuous, the pre-trained V2A model for continuous sound generation is effectively enhanced. Experiments on VGGSoundContinuous and VGGSound show that Smooth Foley has superior generation performance than the baseline model in terms of audio quality, semantics, and temporal alignment.

[0037] like Fig. 9 The figure is a schematic diagram of the structure of a video-audio generation system based on semantic information guidance provided by an embodiment of the present invention. The system can execute the video-audio generation method based on semantic information guidance described in any of the above embodiments and be configured in a terminal.

[0038] The video-audio generation system 10 provided in this embodiment based on semantic information guidance includes: an embedding determination module 11, a similarity determination module 12, a timestamp prediction module 13 and a generation module 14.

[0039] Among them, the embedding determination module 11 is used to input the video frame of the video and the label information of the video used as a guide for detecting the appearance of the sound object into the pre-trained model of comparative learning language-image, so as to obtain the video latent layer representation of the video frame and the textual latent layer representation of the label information; the similarity determination module 12 is used to determine the semantic similarity of the textual latent layer representation and the video latent layer representation; the timestamp prediction module 13 is used to use the semantic similarity as a guide to predict the timestamp of the sound node of the video; the generation module 14 is used to linearly project the video latent layer representation into the audio synthesis model, and use the spatiotemporal information regulator to control the audio synthesis model according to the timestamp to generate audio aligned with the video.

[0040] The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the video-audio generation method based on semantic information guidance in any of the above method embodiments; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: Inputting a video frame of a video and label information of the video used as a guide for detecting the presence of a sound object into a pre-trained model for contrastive learning of language-image, thereby obtaining a video latent layer representation of the video frame and a textual latent layer representation of the label information; Determining the semantic similarity between the text latent representation and the video latent representation; Using the semantic similarity as a guide, predicting the timestamp of the sounding node of the video; The video hidden layer representation is linearly projected into an audio synthesis model, and a spatiotemporal information regulator is used to control the audio synthesis model according to the timestamp to generate audio aligned with the video.

[0041] like Fig.10 The figure shows a structural diagram of a training system for a video-audio generation model based on semantic information guidance provided by one embodiment of the present invention. The system can execute the training method for a video-audio generation model based on semantic information guidance described in any of the above embodiments and be configured in a terminal.

[0042] A training system 20 for a video-audio generation model guided by semantic information provided in this embodiment includes: an input module 21, an embedding determination module 22, a similarity determination module 23, a timestamp prediction module 24, a generation module 25 and a training module 26.

[0043] The input module 21 is used to input the video frame of the video and the label information of the video into the video-audio generation model, wherein the video-audio generation model includes: a pre-trained model of contrastive learning language-image, a visual encoder and a framework adapter for applying the pre-trained model of contrastive learning language-image to video-to-audio conversion, a frame adapter and a spatiotemporal information regulator trained independently, and an audio synthesis model; the embedding determination module 22 is used to obtain the video hidden layer representation of the video frame and the text hidden layer representation of the label information through the encoder of the pre-trained model of contrastive learning language-image; the similarity determination module 23 is used to use the visual encoder and the framework adapter with a projection layer to embed the text into the video frame and the label information; The hidden layer representation and the video hidden layer representation are adapted to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable; the timestamp prediction module 24 is used to use the semantic similarity as a guide to predict the timestamp of the sound node of the video; the generation module 25 is used to use the frame adapter to linearly project the video hidden layer representation into the audio synthesis model, and use the spatiotemporal information regulator to control the audio synthesis model according to the timestamp to generate predicted audio aligned with the video; the training module 26 is used to independently train the frame adapter and the spatiotemporal information regulator based on the diffusion loss determined by the predicted audio until the training requirements are met.

[0044] The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the training method of the video-audio generation model based on semantic information guidance in any of the above method embodiments; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: Inputting video frames of a video and label information of the video into the video-audio generation model, wherein the video-audio generation model includes: a pre-trained model for contrastive learning of language-image, a visual encoder and a frame adapter for applying the pre-trained model for contrastive learning of language-image to video-audio conversion, a frame adapter and a spatiotemporal information regulator that are independently trained, and an audio synthesis model; Obtaining a video latent layer representation of the video frame and a text latent layer representation of the label information through an encoder of the pre-trained model of the comparative learning language-image; Adapting the text hidden layer representation and the video hidden layer representation using a visual encoder and a framework adapter with a projection layer to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable; Using the semantic similarity as a guide, predicting the timestamp of the sounding node of the video; Using the frame adapter to linearly project the video hidden layer representation into the audio synthesis model, and using the spatiotemporal information regulator to control the audio synthesis model according to the timestamp to generate predicted audio aligned with the video; The frame adapter and the spatiotemporal information regulator are trained independently based on the diffusion loss determined by the predicted audio until the training requirements are met.

[0045] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the video-audio generation based on semantic information guidance and the training method of the video-audio generation model based on semantic information guidance in any of the above method embodiments are executed.

[0046] Fig.11 FIG. 1 is a schematic diagram of the hardware structure of an electronic device of a video-audio generation method based on semantic information guidance provided by another embodiment of the present application, such as Fig.11 As shown, the device includes: One or more processors 1110 and memory 1120, Fig.11 A processor 1110 is taken as an example. The apparatus of the video-audio generation method based on semantic information guidance may further include: an input device 1130 and an output device 1140 .

[0047] The processor 1110, the memory 1120, the input device 1130 and the output device 1140 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0048] The memory 1120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the video-audio generation method guided by semantic information in the embodiment of the present application. The processor 1110 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1120, that is, the video-audio generation method guided by semantic information in the above method embodiment is implemented.

[0049] The memory 1120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 1120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1120 may optionally include a memory remotely arranged relative to the processor 1110, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0050] The input device 1130 can receive input digital or character information. The output device 1140 can include a display device such as a display screen.

[0051] The one or more modules are stored in the memory 1120, and when executed by the one or more processors 1110, the video-audio generation method based on semantic information guidance in any of the above method embodiments is executed.

[0052] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0053] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0054] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the video-audio generation method guided by semantic information according to any embodiment of the present invention.

[0055] The electronic device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0056] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

[0057] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0058] (4) Other electronic devices with data processing functions.

[0059] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0060] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0061] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video-audio generation method based on semantic information guidance, comprising: Inputting a video frame of a video and label information of the video used as a guide for detecting the presence of a sound object into a pre-trained model for contrastive learning of language-image, thereby obtaining a video latent layer representation of the video frame and a textual latent layer representation of the label information; Determining the semantic similarity between the text latent representation and the video latent representation; Using the semantic similarity as a guide, predicting the timestamp of the sounding node of the video; The video hidden layer representation is linearly projected into an audio synthesis model, and a spatiotemporal information regulator is used to control the audio synthesis model according to the timestamp to generate audio aligned with the video.

2. The method according to claim 1, wherein: The label information used as a guide for detecting the occurrence of a sound object includes: a plurality of label information of different sound objects in the video.

3. The method according to claim 1, wherein: The semantic similarity is determined by the embedding cosine similarity of the text latent representation and the video latent representation.

4. A training method for a video-audio generation model guided by semantic information, comprising: Inputting video frames of a video and label information of the video into the video-audio generation model, wherein the video-audio generation model includes: a pre-trained model for contrastive learning of language-image, a visual encoder and a frame adapter for applying the pre-trained model for contrastive learning of language-image to video-audio conversion, a frame adapter and a spatiotemporal information regulator that are independently trained, and an audio synthesis model; Obtaining a video latent layer representation of the video frame and a text latent layer representation of the label information through an encoder of the pre-trained model of the comparative learning language-image; Adapting the text hidden layer representation and the video hidden layer representation using a visual encoder and a framework adapter with a projection layer to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable; Using the semantic similarity as a guide, predicting the timestamp of the sounding node of the video; Using the frame adapter to linearly project the video hidden layer representation into the audio synthesis model, and using the spatiotemporal information regulator to control the audio synthesis model according to the timestamp to generate predicted audio aligned with the video; The frame adapter and the spatiotemporal information regulator are trained independently based on the diffusion loss determined by the predicted audio until the training requirements are met.

5. The method according to claim 4, wherein: When training the frame adapter, the projection layer is trained simultaneously, and when training the spatiotemporal information regulator, the audio synthesis model is trained simultaneously.

6. A video-audio generation system guided by semantic information, comprising: An embedding determination module is used to input a video frame of a video and label information of the video used as a guide for detecting the presence of a sound object into a pre-trained model for contrastive learning of language-image, so as to obtain a video latent layer representation of the video frame and a textual latent layer representation of the label information; A similarity determination module, used to determine the semantic similarity between the text hidden layer representation and the video hidden layer representation; A timestamp prediction module, used to predict the timestamp of the sounding node of the video using the semantic similarity as a guide; A generation module is used to linearly project the video hidden layer representation into the audio synthesis model, and use the spatiotemporal information regulator to control the audio synthesis model according to the timestamp to generate audio aligned with the video.

7. The system according to claim 6, wherein: The label information used as a guide for detecting the occurrence of a sound object includes: a plurality of label information of different sound objects in the video.

8. The system according to claim 6, wherein: The semantic similarity is determined by the embedding cosine similarity of the text latent representation and the video latent representation.

9. A training system for a video-audio generation model guided by semantic information, comprising: An input module, used for inputting video frames of a video and label information of the video into the video-audio generation model, wherein the video-audio generation model includes: a pre-trained model for contrastive learning of language-image, a visual encoder and a frame adapter for applying the pre-trained model for contrastive learning of language-image to video-audio conversion, a frame adapter and a spatiotemporal information regulator that are independently trained, and an audio synthesis model; An embedding determination module, configured to obtain a video latent layer representation of the video frame and a text latent layer representation of the label information through an encoder of a pre-trained model of language-image through comparative learning; A similarity determination module is used to adapt the text hidden layer representation and the video hidden layer representation using a visual encoder with a projection layer and a framework adapter to determine the semantic similarity of the text hidden layer representation and the video hidden layer representation, wherein the projection layer is learnable and trainable; A timestamp prediction module, used to predict the timestamp of the sounding node of the video using the semantic similarity as a guide; A generation module, configured to linearly project the video hidden layer representation into an audio synthesis model using the frame adapter, and to control the audio synthesis model to generate predicted audio aligned with the video according to the timestamp using a spatiotemporal information regulator; A training module is used to train the frame adapter and the spatiotemporal information regulator independently based on the diffusion loss determined by the predicted audio until the training requirements are met.

10. The system according to claim 9, wherein: The training module is used to: When training the frame adapter, the projection layer is trained simultaneously, and when training the spatiotemporal information regulator, the audio synthesis model is trained simultaneously.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

12. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 5.

13. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images