Audio and video synthesis method and device based on given multi-modal information
By generating audio that matches the lip movements in the video using 3DCNN, Conformer encoder, and BigVGAN2 decoder, and combining CLIP and DiT models to generate video that matches the audio, the problem of audio and video lip movement mismatch in existing technologies is solved, and multimodal fusion of audio and video and dubbing coordination are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-18
- Publication Date
- 2026-03-13
AI Technical Summary
Existing speech synthesis models cannot adapt to existing videos for dubbing, resulting in significant differences between the synthesized audio and the video lip movements.
A 3DCNN convolutional neural network is used to extract spatiotemporal features from the video to generate conditional embeddings. Audio matching the video lip movements is generated through a Conformer encoder and a BigVGAN2 decoder. At the same time, the CLIP model is used to convert the text into conditional embeddings, and a DiT model is combined to generate a video matching the audio.
It achieves coordinated synthesis of audio and video lip movements, generates video files with coordinated dubbing, and can integrate text information to adapt existing videos for dubbing.
Smart Images
Figure CN121662023A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio and video synthesis technology, and in particular to an audio and video synthesis method and apparatus based on given multimodal information. Background Technology
[0002] Existing speech synthesis models do not reference the lip movements of videos, so they cannot adapt the lip movements of existing videos for dubbing, resulting in a large difference between the synthesized audio and the video lip movements. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide an audio and video synthesis method and apparatus based on given multimodal information, which can better adapt to existing videos for dubbing.
[0004] The technical solution adopted by this invention to solve its technical problem is: to provide an audio and video synthesis method based on given multimodal information, comprising the following steps: Acquire the target video; A feature extraction network is used to extract spatiotemporal features from the target video, and spatiotemporal feature conditional embeddings are generated based on the extracted spatiotemporal features; The spatiotemporal feature conditions are embedded into the audio generation network model to generate audio corresponding to the lip movements of the target video. The audio is combined with the target video.
[0005] The feature extraction network is a 3DCNN convolutional neural network.
[0006] The audio generation network model uses a Conformer encoder and a BigVGAN2 decoder.
[0007] The audio and video synthesis method based on given multimodal information further includes: The CLIP model is used to convert the target text into text conditional embeddings; An encoder is used to encode the audio corresponding to the lip movements of the target video to obtain audio conditional embedding; The target video is segmented into video blocks, and combined with the text conditional embedding and audio conditional embedding, a video matching the audio is generated through a video generation network model. The generated video that matches the audio is then combined with the audio.
[0008] The video generation network model adopts the DiT model.
[0009] The technical solution adopted by this invention to solve its technical problem is: to provide an audio and video synthesis device based on given multimodal information, comprising: The acquisition module is used to acquire the target video; The feature extraction module is used to extract spatiotemporal features from the target video using a feature extraction network, and to generate spatiotemporal feature conditional embeddings based on the extracted spatiotemporal features. The audio generation module is used to embed the spatiotemporal feature conditions into the audio generation network model to generate audio corresponding to the lip movements of the target video. The first synthesis module is used to synthesize the audio with the target video.
[0010] The audio and video synthesis device based on given multimodal information further includes: The conversion module is used to convert target text into text conditional embeddings using the CLIP model. The encoding module is used to encode the audio corresponding to the lip movements of the target video using an encoder to obtain audio conditional embedding; The video generation module is used to segment the target video into video blocks and, in conjunction with the text conditional embedding and audio conditional embedding, generate a video that matches the audio through a video generation network model. The second synthesis module is used to synthesize the generated video and the audio that match the audio.
[0011] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned audio and video synthesis method based on given multimodal information.
[0012] The technical solution adopted by the present invention to solve its technical problem is: to provide a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned audio and video synthesis method based on given multimodal information.
[0013] Beneficial effects By adopting the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art: This invention utilizes 3DCNN to extract spatiotemporal features from videos, generates conditional embeddings, and then uses a Conformer encoder and a BigVGAN2 decoder to generate audio that matches the lip movements in the video. The resulting audio is then combined with the video to obtain a video file with coordinated dubbing.
[0014] This invention can also embed audio and text information that match the lip movements in the video as conditions to generate new video content that is adapted to the original video lip movements and incorporates text information, thus realizing multimodal fusion from video, audio and text. Attached Figure Description
[0015] Figure 1 This is a flowchart of the first embodiment of the audio and video synthesis method based on given multimodal information of the present invention; Figure 2 This is a flowchart of the second embodiment of the audio and video synthesis method based on given multimodal information of the present invention. Detailed Implementation
[0016] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.
[0017] The first embodiment of the present invention relates to an audio and video synthesis method based on given multimodal information, such as... Figure 1 As shown, it includes the following steps: Step 1: Obtain the target video.
[0018] Step 2 involves using a feature extraction network to extract spatiotemporal features from the target video and generating spatiotemporal feature conditional embeddings based on the extracted spatiotemporal features. In this step, the feature extraction network can be a 3D CNN convolutional neural network, trained on a professional-grade video dataset to effectively extract the spatiotemporal features of the target video.
[0019] Step 3: The spatiotemporal feature conditions are embedded into the audio generation network model to generate audio corresponding to the lip movements of the target video. The audio generation network model in this step uses a Conformer encoder and a BigVGAN2 decoder. The Conformer encoder is a hybrid model combining Transformer and CNN for sequence modeling tasks; its encoder part includes self-attention, convolution, and feedforward modules. The BigVGAN2 decoder can employ various decoding strategies (such as Viterbi, CTC, attention, etc.). This audio generation network model can then produce audio that matches the original video lip movements.
[0020] Step 4: Combine the audio with the target video.
[0021] The audio-video synthesis method based on given multimodal information in this embodiment can complete the dubbing of movies. By using 3DCNN to extract spatiotemporal features from the video, a spatiotemporal feature conditional embedding is generated. The spatiotemporal feature conditional embedding is then passed through a Conformer encoder and a BigVGAN2 decoder to generate audio that matches the lip movements of the video. The obtained audio and video are then synthesized to obtain a video file with coordinated dubbing.
[0022] The second embodiment of the present invention also relates to an audio-visual synthesis method based on given multimodal information, which, compared with the first embodiment, such as... Figure 2 As shown, the difference lies in that this embodiment further includes the following after step 3 of the first embodiment: Step 4: Use the CLIP model to convert the target text into text conditional embeddings. The CLIP model in this step can be a CLIP architecture model based on the DiT architecture. During training, it can be trained on a professional-grade text dataset, enabling the CLIP model to output accurate text conditional embeddings.
[0023] Step 5: Encode the audio corresponding to the lip movements of the target video using an encoder to obtain audio conditional embedding. In this embodiment, the encoder used can be a Conformer encoder.
[0024] Step 6: The target video is segmented into video patches, and combined with the text conditional embedding and audio conditional embedding, a video matching the audio is generated through a video generation network model. The video generation network model in this embodiment can employ the DiT model, which segments multi-frame input images (i.e., video) into video patches, converts them into token sequences, and then uses the Transformer module for feature extraction and generation to obtain a video matching the audio.
[0025] Step 7: Combine the generated video that matches the audio with the audio.
[0026] This embodiment of the audio-video synthesis method based on given multimodal information, after obtaining the audio corresponding to the lip movements of the target video, uses text information and audio as conditional embeddings. It then uses a video generation network model to correct the lip movements of the characters in the target video, generating a video that perfectly matches the lip movements of the audio, thereby achieving multimodal fusion from video, audio, to text. This method can be applied to dubbing of translated films, enabling a perfect match between the lip movements of characters in the translated film and the dubbing.
[0027] A third embodiment of the present invention relates to an audio-visual synthesis apparatus based on given multimodal information, comprising: The acquisition module is used to acquire the target video; The feature extraction module is used to extract spatiotemporal features from the target video using a feature extraction network, and to generate spatiotemporal feature conditional embeddings based on the extracted spatiotemporal features. The audio generation module is used to embed the spatiotemporal feature conditions into the audio generation network model to generate audio corresponding to the lip movements of the target video. The first synthesis module is used to synthesize the audio with the target video.
[0028] The feature extraction network is a 3DCNN convolutional neural network.
[0029] The audio generation network model uses a Conformer encoder and a BigVGAN2 decoder.
[0030] The audio and video synthesis device based on given multimodal information further includes: The conversion module is used to convert target text into text conditional embeddings using the CLIP model. The encoding module is used to encode the audio corresponding to the lip movements of the target video using an encoder to obtain audio conditional embedding; The video generation module is used to segment the target video into video blocks and, in conjunction with the text conditional embedding and audio conditional embedding, generate a video that matches the audio through a video generation network model. The second synthesis module is used to synthesize the generated video and the audio that match the audio.
[0031] The video generation network model adopts the DiT model.
[0032] The fourth embodiment of the present invention relates to an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the audio and video synthesis method based on given multimodal information described in the first or second embodiment above.
[0033] The fifth embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the audio-video synthesis method based on given multimodal information described in the first or second embodiment above.
[0034] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0035] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0036] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0037] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0038] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for audio and video synthesis based on given multimodal information, characterized in that, Includes the following steps: Acquire the target video; A feature extraction network is used to extract spatiotemporal features from the target video, and spatiotemporal feature conditional embeddings are generated based on the extracted spatiotemporal features; The spatiotemporal feature conditions are embedded into the audio generation network model to generate audio corresponding to the lip movements of the target video. The audio is combined with the target video.
2. The audio-visual synthesis method based on given multimodal information according to claim 1, characterized in that, The feature extraction network is a 3DCNN convolutional neural network.
3. The audio-visual synthesis method based on given multimodal information according to claim 1, characterized in that, The audio generation network model uses a Conformer encoder and a BigVGAN2 decoder.
4. The audio-visual synthesis method based on given multimodal information according to claim 1, characterized in that, Also includes: The CLIP model is used to convert the target text into text conditional embeddings; An encoder is used to encode the audio corresponding to the lip movements of the target video to obtain audio conditional embedding; The target video is segmented into video blocks, and combined with the text conditional embedding and audio conditional embedding, a video matching the audio is generated through a video generation network model. The generated video that matches the audio is then combined with the audio.
5. The audio-visual synthesis method based on given multimodal information according to claim 1, characterized in that, The video generation network model adopts the DiT model.
6. An audio-visual synthesis device based on given multimodal information, characterized in that, include: The acquisition module is used to acquire the target video; The feature extraction module is used to extract spatiotemporal features from the target video using a feature extraction network, and to generate spatiotemporal feature conditional embeddings based on the extracted spatiotemporal features. The audio generation module is used to embed the spatiotemporal feature conditions into the audio generation network model to generate audio corresponding to the lip movements of the target video. The first synthesis module is used to synthesize the audio with the target video.
7. The audio-visual synthesis apparatus based on given multimodal information according to claim 6, characterized in that, Also includes: The conversion module is used to convert target text into text conditional embeddings using the CLIP model. The encoding module is used to encode the audio corresponding to the lip movements of the target video using an encoder to obtain audio conditional embedding; The video generation module is used to segment the target video into video blocks and, in conjunction with the text conditional embedding and audio conditional embedding, generate a video that matches the audio through a video generation network model. The second synthesis module is used to synthesize the generated video that matches the audio and the audio.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the audio and video synthesis method based on given multimodal information as described in any one of claims 1-5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the audio and video synthesis method based on given multimodal information as described in any one of claims 1-5.