Video generation method, apparatus, computer equipment and storage medium

By extending the image injection and speech adaptation networks of the video generation model, the problem of excessive influence of the textual video model in the denoising stage is solved, realizing the generation of natural, dynamic videos that are adapted to the speech, thus improving the video generation effect.

CN123123653APending Publication Date: 2026-11-17TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510632588.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, the Wensheng video model lacks the ability to process image modal data, which leads to the direct linear integration of the reference image into the noisy image during the denoising stage. This has too much impact, resulting in insufficient dynamism in the generated video content and poor generation effect.

Method used

A video generation model extended with an image injection network and a speech adaptation network is adopted. By extracting and fusing image and speech features, a video adapted to the reference speech is generated, avoiding the use of padding techniques and improving the naturalness and dynamism of the generated video.

Benefits of technology

The generated videos are more natural, more dynamic, and better adapted to the reference audio, thus improving the overall video generation quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123653A_ABST
    Figure CN123123653A_ABST
Patent Text Reader

Abstract

This application discloses a video generation method, apparatus, computer device, and storage medium, belonging to the field of computer technology. The method includes: acquiring a first video generation model, the first video generation model comprising an image injection network, a speech adaptation network, and a video generation network connected sequentially; extracting first image features of a reference image and noise features of a noisy image through the image injection network, fusing the first image features and noise features to obtain second image features, wherein the reference image includes a reference object; extracting speech features of a reference speech through the speech adaptation network, fusing the second image features and speech features to obtain fused features; and generating a video based on the fused features through the video generation network. By using the image injection network and the speech adaptation network, images and speech are transformed into features easily understood by the video generation network, enabling the generation of videos with higher dynamics and better adaptation to the reference speech, thus improving the video generation effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video generation method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of computer technology, the ways to process videos are becoming increasingly diverse. Currently, it is possible to intelligently generate videos based on artificial intelligence technology. For example, by specifying a reference image, a video that matches the content in the reference image can be intelligently generated.

[0003] In related technologies, powerful text-based video models can be used to perform the task of generating video from images. Since text-based video models do not have the ability to process image modal data, padding techniques are used in the denoising stage to directly integrate the reference image into the noisy image in a linear manner to ensure that the generated video matches the reference image.

[0004] However, the above method can lead to the reference image having too much influence on the denoising process, resulting in insufficient dynamism in the final video and thus poor video generation quality. Summary of the Invention

[0005] This application provides a video generation method, apparatus, computer device, and storage medium, which can generate more natural, dynamic videos that are adapted to reference speech, thereby improving the video generation effect. The technical solution is as follows:

[0006] On the one hand, a video generation method is provided, the method comprising:

[0007] Obtain a first video generation model, which includes an image injection network, a speech adaptation network, and a video generation network connected in sequence.

[0008] The image injection network extracts first image features from a reference image and noise features from a noisy image, and fuses the first image features and the noise features to obtain second image features. The reference image includes a reference object.

[0009] The speech features of the reference speech are extracted through the speech adaptation network, and the second image features and the speech features are fused to obtain fused features. The reference speech is the speech of the reference object.

[0010] The video is generated by the video generation network based on the fusion features. The video includes the reference object, and the lip movements of the reference object are adapted to the reference speech.

[0011] On the other hand, a method for training a video generation model is provided, the method comprising:

[0012] A first video generation model is constructed based on an image injection network, a speech adaptation network, and a video generation network, wherein the image injection network, the speech adaptation network, and the video generation network are connected in sequence.

[0013] Acquire a sample reference image, a sample noise image, a sample reference speech, and a labeled video. The sample reference image includes a sample reference object, the labeled video includes the sample reference object, and the lip movements of the sample reference object are adapted to the sample reference speech.

[0014] The image injection network extracts the first sample image features of the sample reference image and the sample noise features of the sample noise image, and then fuses the first sample image features and the sample noise features to obtain the second sample image features.

[0015] The speech features of the sample reference speech are extracted through the speech adaptation network, and the sample image features and the sample speech features are fused to obtain the sample fusion features.

[0016] Sample videos are generated using the video generation network based on the sample fusion features;

[0017] The first video generation model is trained based on the sample video and the labeled video.

[0018] On the other hand, a video generation apparatus is provided, the apparatus comprising:

[0019] The model acquisition module is used to acquire a first video generation model, which includes an image injection network, a speech adaptation network and a video generation network connected in sequence.

[0020] The image processing module is used to extract first image features of a reference image and noise features of a noisy image through the image injection network, and fuse the first image features and the noise features to obtain second image features, wherein the reference image includes a reference object;

[0021] The speech processing module is used to extract speech features of a reference speech through the speech adaptation network, fuse the second image features and the speech features to obtain fused features, wherein the reference speech is the speech of the reference object;

[0022] A video generation module is used to generate a video based on the fusion features through the video generation network, wherein the video includes the reference object and the lip movements of the reference object are adapted to the reference speech.

[0023] Optionally, the reference image has M frames, the noisy image has M frames, where M is an integer greater than 1, and the image injection network includes a first feature extraction layer, a second feature extraction layer, and a feature fusion layer; the image processing module is used for:

[0024] The first feature extraction layer performs three-dimensional convolution on the M reference images to obtain the features of the first N images, where N is a positive integer less than M.

[0025] The second feature extraction layer performs three-dimensional convolution on the M-frame noise images to obtain N-frame noise features.

[0026] The feature fusion layer fuses the first image features of the N frames with the noise features of the N frames to obtain the second image features.

[0027] Optionally, the image injection network includes a first feature fusion layer and a second feature fusion layer, and the image processing module is used for:

[0028] The first feature fusion layer adds the first image features of the N frames to the noise features of the N frames to obtain intermediate image features, wherein the number of frames of the intermediate image features is N frames.

[0029] The second feature fusion layer concatenates the first image feature of the first frame in the N frames of the first image feature with the intermediate image features to obtain the second image feature, and the number of frames of the second image feature is N+1 frames.

[0030] Optionally, the device further includes a mask feature acquisition module for acquiring facial mask features, the facial mask features being used to represent the facial region of the reference object in the reference image;

[0031] The voice processing module is used for:

[0032] Cross-attention processing is performed on the second image features and the speech features to obtain cross features;

[0033] The intermediate feature is obtained by multiplying the intersection feature with the facial mask feature;

[0034] The second image feature and the intermediate feature are added together to obtain the fused feature.

[0035] Optionally, the reference image includes multiple reference objects, the reference speech includes a first reference speech and a second reference speech, the facial mask feature includes a first facial mask feature and a second facial mask feature, the first facial mask feature is used to represent the facial region of a first reference object in the reference image, the second facial mask feature is used to represent the facial region of a second reference object in the reference image, the fusion feature includes a first fusion feature and a second fusion feature, the first fusion feature is obtained based on the second image feature, the speech feature of the first reference speech and the first mask feature, and the second fusion feature is obtained based on the second image feature, the speech feature of the second reference speech and the second mask feature; the video generation module is used to:

[0036] A first video is generated through the video generation network based on the first fusion feature. The first video includes the plurality of reference objects, and the lip movements of the first reference objects are adapted to the first reference speech.

[0037] A second video is generated using the video generation network based on the second fusion feature. The second video includes the plurality of reference objects, and the lip movements of the second reference objects are adapted to the second reference speech.

[0038] Optionally, the speech adaptation network includes K third feature fusion layers, and the video generation network includes K diffusion transformation layers, where K is an integer greater than 1; the speech processing module is used for:

[0039] The first third feature fusion layer fuses the second image features and the speech features to obtain the first third fusion feature;

[0040] The i-th third feature is obtained by fusing the (i-1)-th third feature and the (i-1)-th fourth feature through the i-th third feature fusion layer; where i is a positive integer not greater than K, and the i-th fourth feature is obtained through the i-th diffusion transformation layer.

[0041] Optionally, the voice processing module is used for:

[0042] The second image feature and the speech feature are subjected to cross-attention processing to obtain the first cross feature;

[0043] The first cross feature is multiplied by the facial mask feature to obtain the intermediate feature, which is used to represent the facial region of the reference object in the reference image;

[0044] The second image feature and the intermediate feature are added together to obtain the first third fusion feature.

[0045] Optionally, the voice processing module is used for:

[0046] Cross-attention processing is performed on the (i-1)th third fusion feature and the (i-1)th fourth fusion feature to obtain the ith cross feature;

[0047] The i-th cross feature and the (i-1)-th fourth fusion feature are added together to obtain the i-th third fusion feature.

[0048] Optionally, the video generation network further includes a decoding layer; the video generation module is used for:

[0049] The first diffusion transformation layer is used to denoise the first third fusion feature to obtain the first fourth fusion feature;

[0050] The ith third fusion feature is denoised through the ith diffusion transformation layer to obtain the ith fourth fusion feature;

[0051] The video is obtained by decoding the Kth fourth fusion feature through the decoding layer.

[0052] Optionally, the first video generation model further includes a text processing network, which is connected to the video generation network;

[0053] The device further includes a text processing module for extracting text features of a reference text through the text processing network, wherein the reference text is used to describe the action performed by the reference object;

[0054] The video generation module is used to generate the video through the video generation network based on the fusion features and the text features. The video includes a reference object, and the lip movements of the reference object are adapted to the reference speech. The actions of the reference object are the actions described in the reference text.

[0055] On the other hand, a training apparatus for a video generation model is provided, the apparatus comprising:

[0056] The model building module is used to build a first video generation model based on an image injection network, a speech adaptation network, and a video generation network, wherein the image injection network, the speech adaptation network, and the video generation network are connected in sequence.

[0057] The sample data acquisition module is used to acquire sample reference images, sample noise images, sample reference speech, and labeled videos. The sample reference images include sample reference objects, the labeled videos include the sample reference objects, and the lip movements of the sample reference objects are adapted to the sample reference speech.

[0058] The image processing module is used to extract the first sample image features of the sample reference image and the sample noise features of the sample noise image through the image injection network, and fuse the first sample image features and the sample noise features to obtain the second sample image features;

[0059] The speech processing module is used to extract sample speech features of the sample reference speech through the speech adaptation network, and fuse the second sample image features and the sample speech features to obtain sample fusion features;

[0060] A video generation module is used to generate sample videos based on the sample fusion features through the video generation network;

[0061] The model training module is used to train the first video generation model based on the sample video and the labeled video.

[0062] Optionally, the model building module is further configured to:

[0063] The trained feature extraction layer is copied into a first feature extraction layer and a second feature extraction layer;

[0064] The image injection network is constructed based on the first feature extraction layer, the second feature extraction layer, and the feature fusion layer;

[0065] The first feature extraction layer is used to extract the first sample image features of the sample reference image, and the second feature extraction layer is used to extract the sample noise features of the sample noise image.

[0066] Optionally, the first video generation model further includes the text processing network, which is connected to the video generation network; the device further includes a text processing module, used to extract sample text features of sample reference text through the text processing network, wherein the sample reference text is used to describe the action performed by the reference object in the tagged video;

[0067] The video generation module is used to generate the sample video based on the sample fusion features and the sample text features through the video generation network.

[0068] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the operations performed by the video generation method as described above, or to implement the operations performed by the training method for the video generation model as described above.

[0069] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to implement the operations performed by the video generation method as described above, or to implement the operations performed by the training method for the video generation model as described above.

[0070] On the other hand, a computer program product is provided, including a computer program loaded and executed by a processor to implement the operations performed by the video generation method as described above, or to implement the operations performed by the training method for the video generation model as described above.

[0071] The methods, apparatus, computer devices, and storage media provided in this application embodiment include a first video generation model comprising an image injection network, a speech adaptation network, and a video generation network. This model provides the ability to process image and speech modal data and to generate videos. The image injection network and speech adaptation network can transform images and speech into features easily understood by the video generation network. The image injection network acquires image features based on the image, and the speech adaptation network acquires speech features based on the speech. These image and speech features are then fused to obtain fused features. The video generation network then generates videos based on these fused features. This avoids the use of padding techniques to introduce images during the denoising stage, resulting in more natural, dynamic videos that are adapted to the reference speech, thus improving the video generation effect. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application;

[0074] Figure 2 This is a flowchart of a video generation method provided in an embodiment of this application;

[0075] Figure 3 This is a flowchart of another video generation method provided in the embodiments of this application;

[0076] Figure 4 This is a schematic diagram of the structure of a first video generation model provided in an embodiment of this application;

[0077] Figure 5This is a flowchart of another video generation method provided in the embodiments of this application;

[0078] Figure 6 This is a flowchart of another video generation method provided in the embodiments of this application;

[0079] Figure 7 This is a schematic diagram of another video generation method provided in an embodiment of this application;

[0080] Figure 8 This is a schematic diagram of another video generation method provided in an embodiment of this application;

[0081] Figure 9 This is a flowchart of a training method for a video generation model provided in an embodiment of this application;

[0082] Figure 10 This is a schematic diagram of a video filtering method provided in an embodiment of this application;

[0083] Figure 11 This is a flowchart of another training method for a video generation model provided in an embodiment of this application;

[0084] Figure 12 This is a flowchart of another training method for a video generation model provided in an embodiment of this application;

[0085] Figure 13 This is a flowchart of a voice-driven virtual character video generation method provided in an embodiment of this application;

[0086] Figure 14 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application;

[0087] Figure 15 This is a schematic diagram of another video generation device provided in an embodiment of this application;

[0088] Figure 16 This is a schematic diagram of the structure of a training device for a video generation model provided in an embodiment of this application;

[0089] Figure 17 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0090] Figure 18 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0092] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first video generation model may be referred to as a second video generation model, and similarly, a second video generation model may be referred to as a first video generation model.

[0093] "At least one" refers to one or more images. For example, at least one image can be one image, two images, three images, or any integer number of images greater than or equal to one. "Multiple" refers to two or more images. For example, multiple images can be two images, three images, or any integer number of images greater than or equal to two. "Each" refers to each of the at least one images. For example, each image refers to each of the multiple images. If the multiple images consist of three images, then each image refers to each of the three images.

[0094] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have all been fully authorized by the user or relevant parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0095] For example, the reference images, reference audio, reference text, etc. involved in this application are all fully authorized by the user or relevant parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0096] For ease of understanding, the concepts involved in the embodiments of this application are explained below.

[0097] (1) Voice-driven character animation generation: Based on the reference image and reference voice, generate a character animation video that is consistent with the character in the reference image and adapted to the reference voice.

[0098] (2) DiT (Diffusion Transformer): A diffusion model based on the Transformer architecture that generates high-quality images through an iterative denoising process.

[0099] (3)MMDiT (MultiModal Diffusion Transformer): A diffusion model for processing multimodal data, including images, speech and text.

[0100] (4) 3D Casual VAE (3D Causal Variational Autoencoder): A neural network used to compress video into latent vectors.

[0101] (5) Cross-attention: A mechanism for establishing associations between different input data in a model, which weights and integrates the features of one input data into the features of another input data.

[0102] The video generation method provided in this application can be used in computer devices. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. This application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0103] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application. See also... Figure 1 The computer system includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0104] Terminal 101 has a client 111 installed and running. This client 111 can be a social application client, online payment client, online shopping client, game client, medical service client, video client, etc. When terminal 101 runs client 111, the user interface of client 111 is displayed on the screen of terminal 101. Terminal 101 is the terminal used by user 121.

[0105] Optionally, terminal 101 can refer to one of a plurality of terminals. As those skilled in the art will know, the number of terminals can be more or less. For example, there may be only one terminal, or there may be six, eight or more terminals. This application does not limit the number of terminals or the type of device.

[0106] Figure 1Only one terminal is shown in the diagram, but in different embodiments, multiple other terminals 103 can access the server 102. Optionally, one or more terminals 103 may also be terminals corresponding to developers, on which a client development and editing platform is installed. Developers can edit and update the client on the terminal 103 and transmit the updated client installation package to the server 102 via wired or wireless network. Terminal 101 can download the client installation package from the server 102 to update the client.

[0107] Terminal 101 and other terminals 103 are connected to server 102 via wired or wireless networks.

[0108] Server 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 102 is used to provide backend services to clients. Optionally, server 102 undertakes the primary computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the primary computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0109] In this embodiment, terminal 101 receives a reference image and reference audio uploaded by a user, and sends a video generation request to server 102, the video generation request carrying the reference image and reference audio. Upon receiving the video generation request, server 102 uses the method provided in this embodiment to generate a video based on the reference image and reference audio. The video includes a reference object in the reference image, and the lip movements of the reference object match the reference audio. Server 102 then returns the video to terminal 101.

[0110] It should be noted that the above implementation environment is only an example. The method provided in this application embodiment can also be executed by terminal 101 or server 102 alone, or by other computer devices. This application embodiment does not limit this.

[0111] In related technologies, for voice-driven human animation generation tasks, high-performance text-based video models are typically used to perform image-to-video generation. Since text-based video models lack the ability to process image modal data, padding techniques are employed in the denoising stage to directly and linearly integrate the reference image into the noisy image to ensure the generated video matches the reference image. However, this approach leads to the reference image having an excessive influence on the denoising process, resulting in insufficient dynamism in the final video and poor video generation quality. Alternatively, a text-based video model can be pre-trained. However, since text-based video models are usually derived by modifying the underlying model of a text-based video model, the modified underlying model reduces the video generation capability, making the text-based video model unstable and prone to distortion in the generated video.

[0112] Figure 2 This is a flowchart of a video generation method provided in an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0113] 1. A first video generation model is constructed based on an image injection network, a speech adaptation network, and a video generation network. The video generation network can be a network obtained from a second video generation model, which is a pre-trained model. This first video generation model, while retaining the video generation capabilities of the second model, extends the data processing capabilities for both image and speech modalities, and can be used to generate video objects based on speech-driven processes. For example, the second video generation model can be a model for generating video based on text.

[0114] 2. Obtain reference image, reference speech, and facial mask features. The reference image includes a reference object, the reference speech is the speech of the reference object, and the facial mask features represent the facial region of the reference object in the reference image.

[0115] 3. The reference image and the noisy image are input into the image injection network of the first video generation model to obtain the second image features. The image injection network realizes the transformation of images into features that are easy for the video generation network to understand.

[0116] 4. The reference speech, facial mask features, and second image features are input into the speech adaptation network of the first video generation model to obtain fused features. The speech adaptation network converts speech into features that are easy for the video generation network to understand and adapts the lip movements of the object to the speech.

[0117] 5. The fused features are input into the video generation network of the first video generation model to obtain a video. The video includes a reference object, and the lip movements of the reference object are adapted to the reference speech, thereby realizing the generation of object videos based on speech-driven generation.

[0118] In this embodiment, without altering the network structure of the video generation network in the second video generation model, an image injection network and a speech adaptation network are additionally extended. The image injection network transforms the image into features easily understood by the video generation network, and the speech adaptation network transforms the speech into features easily understood by the video generation network, adapting the object to the speech. The video generation network then generates a video adapted to both the image and speech. Therefore, the video generation capability of the second video generation model is effectively utilized, enabling the use of speech to drive lip movements of reference objects. This generates natural, highly dynamic object videos that are adapted to the speech, improving the controllability, stability, and effectiveness of video generation.

[0119] Figure 3 This is a flowchart of another video generation method provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 3 The method includes:

[0120] 301. The computer device acquires a first video generation model, which includes an image injection network, a speech adaptation network, and a video generation network connected in sequence.

[0121] The image injection network is connected to the speech adaptation network, and the output data of the image injection network is the input data of the speech adaptation network. The speech adaptation network is connected to the video generation network, and the output data of the speech adaptation network is the input data of the video generation network.

[0122] An image injection network is used to acquire image features based on an image. Specifically, the input data of the image injection network is a reference image and a noisy image, and the output data of the image injection network is image features. Acquiring image features based on an image includes extracting image features and fusing image features. This image injection network is used to perform the operation of feature extraction and the operation of fusing image features.

[0123] The speech adaptation network is used to extract speech features based on speech and fuse these speech features with the features output by the image injection network. Specifically, the input data of the speech adaptation network consists of speech and the output data (i.e., image features) of the image injection network, and the output data of the speech adaptation network is the fused feature. The speech adaptation network is used to extract features from the input speech to obtain speech features, and it is also used to fuse the extracted speech features with the image features output by the image injection network to obtain the fused feature.

[0124] The video generation network is used to generate videos based on features output by the speech adaptation network. Specifically, the input data of this video generation network is the output data (i.e., fused features) of the speech adaptation network, and the output data is a video. The video generation network is used to generate videos based on the fused features output by the speech adaptation network.

[0125] In one possible implementation, the video generation network is a network obtained from a second video generation model, which is a trained model.

[0126] In this embodiment, the first video generation model is used to generate video based on images and speech. The images include objects, and the speech drives the lip movements of the objects, so that the generated video includes the objects in the images, and the lip movements of the objects are adapted to the speech. The second video generation model can be any type of video generation model, such as a mixed-mode model that generates video based on text. The video generation network in the first video generation model reuses the video generation network in the second video generation model, thus retaining the video generation capabilities of the second video generation model. The first video generation model is obtained by adding an image injection network and a speech adaptation network to the video generation network. The image injection network can process image data, and the speech adaptation network can process speech data, thereby expanding the data processing capabilities for both image and speech modalities.

[0127] 302. The computer device extracts the first image features of the reference image and the noise features of the noisy image through an image injection network, and fuses the first image features and the noise features to obtain the second image features. The reference image includes a reference object.

[0128] The computer equipment acquires a reference image and a noise image. The reference image is used to indicate reference objects included in the video to be generated. These reference objects can be real people, virtual people, or virtual cartoon characters. For example, the virtual person can be a digital human, which is a digitally created human-like figure produced using digital technology. The noise image can be a white noise image, etc.

[0129] The computer device inputs a reference image and a noisy image into an image injection network. The image injection network performs the following steps: extracting the first image features of the reference image and the noise features of the noisy image, and fusing the first image features and the noise features to obtain the second image features.

[0130] 303. The computer device extracts the speech features of the reference speech through the speech adaptation network, fuses the second image features and speech features to obtain fused features, where the reference speech is the speech of the reference object.

[0131] The computer device acquires reference speech, which is the speech of a reference object. The reference speech can be real speech or speech generated through artificial intelligence technology. In this application embodiment, the purpose of providing reference speech in the video generation task is to ensure that the lip movements of the reference object in the generated video match the reference speech, thereby presenting the effect of the reference object reading the reference speech.

[0132] The computer device inputs reference speech and second image features into the speech adaptation network, and performs the following steps in the speech adaptation network: extracting speech features from the reference speech, fusing the second image features and speech features to obtain fused features.

[0133] 304. A computer device generates a video through a video generation network based on fusion features. The video includes a reference object, and the lip movements of the reference object are adapted to the reference speech.

[0134] A computer device inputs fused features into a video generation network, which then performs a step of generating a video based on these features. The video includes a reference object from a reference image, and the lip movements of this reference object in the video match the lip movements of the reference speech. This matching means that the lip movements of the reference object are consistent with the lip movements when the reference speech is read aloud. Therefore, the video presents the effect of the reference object reading the reference speech, achieving speech-driven object video generation.

[0135] The method provided in this application embodiment includes a first video generation model comprising an image injection network, a speech adaptation network, and a video generation network. It provides the ability to process image and speech modal data and generate videos. The image injection network and speech adaptation network can transform images and speech into features easily understood by the video generation network. The image injection network acquires image features based on the image, and the speech adaptation network acquires speech features based on the speech. The image and speech features are then fused to obtain fused features. The video generation network then generates videos based on these fused features. This avoids the use of padding techniques to introduce images during the denoising stage, resulting in more natural, dynamic videos that are adapted to the reference speech, thus improving the video generation effect.

[0136] Figure 4 This is a schematic diagram of the structure of a first video generation model provided in an embodiment of this application, such as... Figure 4As shown, the first video generation model includes an image injection network 401, a speech adaptation network 402, and a video generation network 403. The image injection network 401 is connected to the speech adaptation network 402, and the output data of the image injection network 401 serves as the input data of the speech adaptation network 402. The speech adaptation network 402 is connected to the video generation network 403, and the output data of the speech adaptation network 402 serves as the input data of the video generation network 403. The image injection network 401 is used to acquire image features based on images, the speech adaptation network 402 is used to acquire speech features based on speech and fuse these speech features with the features output by the image injection network, and the video generation network 403 is used to generate video based on the features output by the speech adaptation network.

[0137] In one possible implementation, such as Figure 4 As shown, the image injection network 401 includes a first feature extraction layer, a second feature extraction layer, a first feature fusion layer, and a second feature fusion layer. The speech adaptation network 402 includes a third feature extraction layer and K third feature fusion layers. The video generation network 403 includes K diffusion conversion layers and a decoding layer, where K is an integer greater than 1.

[0138] The connection relationships of each network layer in the image injection network 401 are as follows: the first feature extraction layer is connected to the first feature fusion layer; the second feature extraction layer is connected to the first feature fusion layer; the input data of the first feature fusion layer is the output data of the first feature extraction layer and the output data of the second feature extraction layer; the second feature fusion layer is connected to both the first feature extraction layer and the first feature fusion layer; the input data of the second feature fusion layer is the output data of the first feature extraction layer and the output data of the first feature fusion layer. The second feature fusion layer is connected to the first third feature fusion layer in the speech adaptation network 402. The functions of each network layer in the image injection network 401 are as follows: the first feature extraction layer is used to extract the first image features of the reference image; the second feature extraction layer is used to extract the noise features of the noisy image; the first feature fusion layer is used to add the first image features output by the first feature extraction layer and the noise features output by the second feature extraction layer to obtain intermediate image features; the second feature fusion layer is used to concatenate the first image features output by the first feature extraction layer and the intermediate image features output by the first feature fusion layer to obtain second image features.

[0139] The connection relationships of each network layer in the speech adaptation network 402 are as follows: the third feature extraction layer is connected to the first third feature fusion layer, and K third feature fusion layers are connected sequentially. The first third feature fusion layer is also connected to the first diffusion transformation layer in the video generation network 403, and the i-th third feature fusion layer is also connected to the (i-1)-th and i-th diffusion transformation layers in the video generation network 403, where i is an integer greater than 1 and not greater than K. The functions of each network layer in the speech adaptation network 402 are as follows: the third feature extraction layer is used to extract speech features from the reference speech; the first third feature fusion layer is used to perform cross-attention processing, multiplication, and addition operations. Specifically, the first third feature fusion layer performs cross-attention processing on the speech features output by the third feature extraction layer and the second image features output by the second feature fusion layer to obtain cross features; multiplies the cross features with the facial mask features to obtain intermediate features; and adds the second image features output by the second feature fusion layer and the intermediate features to obtain fused features. The i-th third feature fusion layer is used to perform cross-attention processing and addition operations. Specifically, the i-th third feature fusion layer is used to perform cross-attention processing on the third fusion feature output by the (i-1)-th third feature fusion layer and the fourth fusion feature output by the (i-1)-th diffusion transformation layer to obtain a cross feature. The cross feature is then added to the fourth fusion feature output by the (i-1)-th diffusion transformation layer to obtain the third fusion feature.

[0140] The connection relationships of the network layers in the video generation network 403 are as follows: K diffusion transformation layers are connected sequentially, and the Kth diffusion transformation layer is connected to the decoding layer. The tth diffusion transformation layer is also connected to the tth third feature fusion layer and the (t+1)th third feature fusion layer in the video generation network 403, where t is a positive integer less than K. The functions of each network layer in the video generation network 403 are as follows: The diffusion transformation layer is used to perform denoising operations. Specifically, the tth diffusion transformation layer is used to denoise the third fused feature output by the tth third feature fusion layer to obtain the fourth fused feature. The decoding layer is used to perform decoding operations. Specifically, the decoding layer is used to decode the fourth fused feature output by the Kth diffusion transformation layer to obtain the video.

[0141] exist Figure 4 Based on the first video generation model shown, the data flow during video generation is as follows: reference image 11 is input to the first feature extraction layer in image injection network 401; noisy image 22 is input to the second feature extraction layer in image injection network 401; reference speech 33 is input to the third feature extraction layer in speech adaptation network 402; and facial mask features 44 are input to the first third feature fusion layer. All of the above data are processed by the first video generation model, and finally, the decoding layer in video generation network 403 outputs video 55.

[0142] The first feature extraction layer extracts the first image features from the reference image 11 and inputs these features into the first feature fusion layer and the second feature fusion layer. The second feature extraction layer extracts the noise features from the noisy image 22 and inputs these noise features into the first feature fusion layer. The first feature fusion layer adds the first image features and the noise features to obtain intermediate image features, which are then input into the second feature fusion layer. The second feature fusion layer concatenates the first image features from the first frame with the intermediate image features to obtain second image features, which are then input into the first third feature fusion layer in the speech adaptation network 402.

[0143] The third feature extraction layer extracts speech features from reference speech 33 and inputs these features into the first third feature fusion layer. The first third feature fusion layer processes the second image features, speech features, and facial mask features 44 to obtain the first third fused feature, which is then input into the first diffusion transformation layer and the second third feature fusion layer. The first diffusion transformation layer denoises the first third fused feature to obtain the first fourth fused feature, which is then input into the second third feature fusion layer. The second third feature fusion layer processes the first and first fourth fused features to obtain the second third fused feature, which is then input into the second diffusion transformation layer and the third third feature fusion layer. This process continues until the Kth diffusion layer obtains the Kth fourth fused feature, which is then input into the decoding layer. The decoding layer decodes the Kth fourth fused feature to obtain the video.

[0144] Figure 5 This is a flowchart of another video generation method provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 5 The method includes:

[0145] 501. The computer device acquires a first video generation model, which includes an image injection network, a speech adaptation network, and a video generation network connected in sequence.

[0146] The first video generation model can be an MMDiT model, and its network structure is as follows: Figure 4 As shown.

[0147] 502. The computer device acquires a reference image, a noisy image, a reference speech, and facial mask features. The reference image includes a reference object, the reference speech is the speech of the reference object, and the facial mask features are used to represent the facial region of the reference object in the reference image.

[0148] The reference object can be a digital human, which is a digitally created human figure that closely resembles a human. This application embodiment can generate a video including a digital human based on voice-driven technology, so that the digital human's lip movements match the reference voice, thereby presenting the effect of the digital human broadcasting the reference voice. Of course, the reference object can also be a real person or other cartoon character, etc., and this application embodiment does not limit the type of reference object.

[0149] The noise image can be a white noise image, and the size of the noise image is the same as the size of the reference image.

[0150] The reference speech is the speech of a reference object, which can be real speech or speech generated through artificial intelligence technology.

[0151] The facial mask feature can be a feature map, the size of which is the same as the size of the reference image. The feature map includes values ​​1 and 0. The region containing the value 1 represents the facial region of the reference object in the reference image, and the region containing the value 0 represents the region in the reference image other than the facial region of the reference object. In one possible implementation, the reference image comprises M frames, each frame including the reference object. However, the position of the facial region of the reference object in different frames may be the same or different. The facial region represented by the facial mask feature can be the union of the facial regions of the reference object in multiple reference images.

[0152] 503. The computer device extracts the first image features of the reference image and the noise features of the noisy image through the image injection network, and fuses the first image features and the noise features to obtain the second image features.

[0153] The computer device inputs a reference image and a noisy image into the image injection network in the first video generation model. The image injection network extracts the first image features of the reference image and the noise features of the noisy image, and fuses the first image features and the noise features to obtain the second image features.

[0154] In one possible implementation, the reference image has M frames, and the noisy image has M frames, where M is an integer greater than 1. For example... Figure 4 As shown, the image injection network includes a first feature extraction layer, a second feature extraction layer, and a feature fusion layer, with the first and second feature extraction layers connected to the feature fusion layer. Therefore, step 502 includes steps 5021-5023.

[0155] 5021. Through the first feature extraction layer, perform three-dimensional convolution on the M frame reference images to obtain the first image features of N frames, where N is a positive integer less than M.

[0156] The first feature extraction layer performs a 3D convolution operation. The computer device inputs M frames of reference images into the first feature extraction layer to obtain N frames of first image features output by the first feature extraction layer. These first image features can be feature maps.

[0157] 3D convolution is an extension of 2D convolution, adding a third dimension. Traditional 2D convolution processes two-dimensional data, typically used for image height and width. 3D convolution adds a depth dimension, processing three-dimensional data, typically used for video or volumetric data. 3D convolution works by sliding the convolution kernel across three dimensions (height, width, and depth). Specifically, the 3D convolution kernel slides across the input data (usually multiple video frames or volumetric data), performing element-wise multiplication and addition operations at each location to generate an output feature map. 3D convolution operations consider not only spatial variations but also temporal or depth variations, capturing more dynamic information. In this embodiment, the reference image is two-dimensional data, and the M-frame reference images are three-dimensional data.

[0158] Optionally, the first feature extraction layer includes a convolutional layer and a fully connected layer. The M-frame reference images are tokenized by the convolutional layer, and the tokenized features are mapped by the fully connected layer to obtain the features of the N-frame first image.

[0159] For example, the convolutional layer may include a 3D Casual VAE network structure or a 3D CNN network structure.

[0160] 5022. Through the second feature extraction layer, perform three-dimensional convolution on the M-frame noise images to obtain N-frame noise features.

[0161] The second feature extraction layer performs a 3D convolution operation. The computer device inputs M frames of noisy images into the second feature extraction layer to obtain N frames of noise features output by the second feature extraction layer. These noise features can be feature maps.

[0162] Optionally, the second feature extraction layer is a convolutional layer and a fully connected layer. The M-frame noisy images are tokenized by the convolutional layer to obtain the N-frame noisy features.

[0163] For example, the convolutional layer may include a 3D Casual VAE network structure or a 3D CNN network structure.

[0164] 5023. Through the feature fusion layer, the first image features of N frames are fused with the noise features of N frames to obtain the second image features.

[0165] The feature fusion layer is used to fuse the input features. The computer device inputs the first image features from N frames and the noise features from N frames into the feature fusion layer to obtain the second image features output by the feature fusion layer.

[0166] Optionally, such as Figure 4 As shown, the image injection network includes a first feature fusion layer and a second feature fusion layer. The computer device uses the first feature fusion layer to add N frames of first image features to N frames of noise features to obtain intermediate image features, with N frames representing the intermediate image features. Then, using the second feature fusion layer, the first frame of the first image features from the N frames of first image features is concatenated with the intermediate image features to obtain second image features, with N+1 frames representing the second image features.

[0167] In this system, both the first image feature and the noise feature are feature maps. The computer device adds the first image feature of N frames and the noise feature of N frames pixel by pixel to obtain the intermediate image feature of N frames. For example, the first image feature of the first frame and the noise feature of the first frame are added to obtain the intermediate image feature of the first frame, the first image feature of the second frame and the noise feature of the second frame are added to obtain the intermediate image feature of the second frame, and so on.

[0168] The computer device concatenates the first image feature of the first frame with the intermediate image features of N frames along the frame dimension to obtain the second image feature. The number of frames for the second image feature is N+1 frames, that is, the second image feature includes the intermediate image features of N frames and the first image feature of 1 frame.

[0169] In this implementation, the first feature extraction layer and the second feature extraction layer can perform three-dimensional convolution on multiple reference images and multiple noisy images. Since three-dimensional convolution can be performed on the height, width and number of frames of the image, the height and width of the image are spatial dimensions and the number of frames of the image is temporal dimension. Therefore, it can capture the features of the image in both the temporal and spatial dimensions at the same time, thereby preserving the dynamic change information of multiple frames in time sequence, which is beneficial to improving the ability to express image features.

[0170] Furthermore, after adding the first image features of N frames and the noise features of N frames to obtain the intermediate image features, the intermediate image features are then concatenated with the first image features of the first frame to obtain the second image features. This preserves the first image features without added noise, avoiding the loss of detailed information from the reference image. This allows the detailed information from the reference image to be effectively injected into each frame of the video generation process, enhancing the temporal consistency of image features and providing more precise control, thereby improving the stability and dynamism of video generation.

[0171] 504. The computer device extracts the speech features of the reference speech through the speech adaptation network, performs cross-attention processing on the second image features and speech features to obtain cross features, multiplies the cross features with the facial mask features to obtain intermediate features, and adds the intermediate features with the second image features to obtain fused features.

[0172] The computer device inputs reference speech, facial mask features, and second image features into the speech adaptation network in the first video generation model. The speech adaptation network extracts the speech features of the reference speech, performs cross-attention processing on the second image features and speech features to obtain cross features, and multiplies the cross features with the facial mask features to obtain fused features.

[0173] In this embodiment, reference speech is introduced to drive the lip movements of the reference object, enabling speech features to interact with image features across modalities during the generation of the reference object's facial movements. This is beneficial for improving the synchronization and naturalness of the reference object's facial lip movements with the reference speech in the final generated video.

[0174] Furthermore, since the reference speech is used to guide the lip movements of the reference object, facial mask features are introduced to indicate the facial region of the reference object. The cross features after cross-attention processing are multiplied with the facial mask features, so that the speech features of the reference speech only act on the facial region of the reference object, avoiding the reference speech from affecting the non-facial region of the reference object. Therefore, the control of the reference object during video generation is more precise, thereby improving the stability and dynamism of video generation.

[0175] In one possible implementation, a sliding window method is used to generate video based on reference speech. The computer device determines the sliding window and its sliding length. The sliding window represents the number of audio frames used to generate a video segment, and the sliding length represents the number of frames the sliding window slides forward to generate the next video segment. During video generation, the computer device acquires a speech segment from the reference speech based on the sliding window. The number of audio frames in this speech segment is equal to the number of audio frames indicated by the sliding window. A video segment is first generated based on this speech segment. Then, the sliding window slides forward in the reference speech according to the sliding length, hitting the next speech segment. The next video segment is generated based on the next speech segment, and so on, thus generating multiple consecutive video segments to form a complete video.

[0176] The length of the sliding window is greater than the sliding length, so there is an overlap between adjacent audio segments. For example, the starting audio frame of the current audio segment is the same as the ending audio frame of the previous audio segment, so as to ensure the continuity between the generated video segments.

[0177] 505. A computer device generates a video through a video generation network based on fusion features. The video includes a reference object, and the lip movements of the reference object are adapted to the reference speech.

[0178] The computer device inputs the fused features into the video generation network of the first video generation model, which then generates a video based on the fused features. This video includes a reference object from a reference image, and the lip movements of the reference object in the video match the reference speech, thus presenting the effect of the reference object reading the reference speech, achieving speech-driven object video generation.

[0179] In one possible implementation, such as Figure 4 As shown, the speech adaptation network includes a third feature extraction layer and K third feature fusion layers, and the video generation network includes K diffusion conversion layers and a decoding layer, where K is an integer greater than 1. Steps 504-505 above can be replaced by steps 1-6:

[0180] 1. Extract speech features of the reference speech through the third feature extraction layer in the speech adaptation network.

[0181] Optionally, the third feature extraction layer may include a Whisper model network structure. The Whisper model is used to convert speech into features.

[0182] 2. By fusing the second image features and speech features through the first third feature fusion layer in the speech adaptation network, the first third fusion feature is obtained.

[0183] The computer device inputs the second image features and speech features into the first third feature fusion layer, which then fuses the second image features and speech features to obtain the first third fused feature.

[0184] Optionally, the computer device performs cross-attention processing on the second image features and speech features through the first third feature fusion layer to obtain the first cross feature, multiplies the first cross feature with the facial mask feature to obtain the intermediate feature, and adds the second image feature and the intermediate feature to obtain the first third fusion feature.

[0185] The first third feature fusion layer includes a cross-attention layer, a multiplication operator, and an addition operator. The cross-attention layer is used to perform cross-attention processing, the multiplication operator is used to perform multiplication operations, and the addition operator is used to perform addition operations.

[0186] In this implementation, facial mask features are introduced to indicate the facial regions of the reference object. The cross-attention processed cross-features are multiplied by these facial mask features, ensuring that the speech features of the reference speech only affect the facial regions of the reference object, preventing the reference speech from influencing non-facial regions. This results in more precise control over the reference object during video generation. Furthermore, after multiplying the cross-features by the facial mask features, they are added to the second image features. This ensures that the complete second image features are preserved in the first third feature fusion layer, avoiding the loss of global features of the second image.

[0187] 3. The first third fusion feature is denoised through the first diffusion transformation layer in the video generation network to obtain the first fourth fusion feature.

[0188] The computer device inputs the first third fusion feature into the first diffusion conversion layer, and the first diffusion conversion layer denoises the first third fusion feature to obtain the first fourth fusion feature.

[0189] Optionally, the first diffusion transformation layer may include the network structure of the DiT model.

[0190] 4. By fusing the (i-1)th third fusion feature and the (i-1)th fourth fusion feature through the i-th third feature fusion layer in the speech adaptation network, the i-th third fusion feature is obtained; where i is a positive integer not greater than K, and the i-th fourth fusion feature is obtained through the i-th diffusion transformation layer.

[0191] The computer device inputs the first and first fourth fusion features into the second third feature fusion layer, and the second third feature fusion layer outputs the second third fusion feature. The computer device then inputs the second and second fourth fusion features into the third third feature fusion layer, and the third third feature fusion layer outputs the third third fusion feature. This process continues until the Kth third feature fusion layer outputs the Kth third fusion feature.

[0192] Optionally, the computer device performs cross-attention processing on the (i-1)th third fusion feature and the (i-1)th fourth fusion feature through the i-th third feature fusion layer to obtain the i-th cross feature, and adds the i-th cross feature and the (i-1)th fourth fusion feature to obtain the i-th third fusion feature.

[0193] The i-th third feature fusion layer includes a cross-attention layer and an addition operator. The cross-attention layer is used to perform cross-attention processing, and the addition operator is used to perform addition operations.

[0194] In this implementation, the (i-1)th third fusion feature is the feature before denoising, and the (i-1)th fourth fusion feature is the feature after denoising. By performing cross-attention processing on the (i-1)th third fusion feature and the (i-1)th fourth fusion feature, the guiding role of speech features in the image generation process can be further strengthened, avoiding the loss of detailed information of speech features during denoising. Furthermore, adding the cross-attention processed i-th cross-feature and the (i-1)th fourth fusion feature can further preserve the denoising result, preventing the influence of the (i-1)th third fusion feature from being too large, which is beneficial for accelerating inference speed.

[0195] 5. The i-th third fusion feature is denoised through the i-th diffusion transformation layer in the video generation network to obtain the i-th fourth fusion feature.

[0196] The computer device inputs the second third fused feature into the second diffusion conversion layer, which denoises the second third fused feature to obtain the second fourth fused feature. The third third fused feature is then input into the third diffusion conversion layer, which denoises the third third fused feature to obtain the third fourth fused feature. This process continues until the Kth diffusion conversion layer outputs the Kth fourth fused feature.

[0197] Optionally, the i-th diffusion transformation layer may include the network structure of the DiT model.

[0198] 6. The Kth fourth fusion feature is decoded through the decoding layer in the video generation network to obtain the video.

[0199] The computer device inputs the Kth fourth fusion feature to the decoding layer, which then decodes the Kth fourth fusion feature to obtain the video.

[0200] In this implementation, the speech adaptation network has multiple third feature fusion layers. In the first third feature fusion layer, speech features and image features are initially fused to guide the lip movements of the reference object in the image. In each of the remaining third feature fusion layers, the third fusion feature output from the previous third feature fusion layer is iteratively fused with the fourth fusion feature output from the diffusion transform layer. This process extracts and enhances the expressive power of cross-modal features layer by layer, which helps to improve the naturalness of image-speech adaptation.

[0201] Furthermore, by setting up multiple diffusion transformation layers in the video generation network and iteratively performing the denoising step on the fused features, the noisy features are transformed into clearer and more stable features through layer-by-layer denoising, thereby providing high-quality features for the decoding process and improving the stability of video generation.

[0202] The method provided in this application embodiment includes a first video generation model comprising an image injection network, a speech adaptation network, and a video generation network. It provides the ability to process image and speech modal data and generate videos. The image injection network and speech adaptation network can transform images and speech into features easily understood by the video generation network. The image injection network acquires image features based on the image, and the speech adaptation network acquires speech features based on the speech. The image and speech features are then fused to obtain fused features. The video generation network then generates videos based on these fused features. This avoids the use of padding techniques to introduce images during the denoising stage, resulting in more natural, dynamic videos that are adapted to the reference speech, thus improving the video generation effect.

[0203] The above Figure 5 The embodiments illustrate the process of generating video based on the lip movements of an object driven by a single voice. In another embodiment, the method provided in this application also supports generating video based on the lip movements of different objects driven by multiple voices; detailed process is described below. Figure 6 Examples of implementations. Figure 6 This is a flowchart of another video generation method provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 6 The method includes:

[0204] 601. A computer device acquires a first video generation model, which includes an image injection network, a speech adaptation network, and a video generation network connected in sequence. The image injection network is used to convert images into features, the speech adaptation network is used to adapt the lip movements of objects to speech, and the video generation network is used to generate videos based on the input features. The video generation network is a network acquired from a trained second video generation model.

[0205] The process of step 601 is the same as that of step 501 above, and will not be repeated here.

[0206] 602. A computer device acquires a reference image, a noisy image, a first reference speech, a second reference speech, a first facial mask feature, and a second facial mask feature. The reference image includes multiple reference objects, including a first reference object and a second reference object. The first reference speech is the speech of the first reference object, and the second reference speech is the speech of the second reference object. The first facial mask feature is used to represent the facial region of the first reference object in the reference image, and the second facial mask feature is used to represent the facial region of the second reference object in the reference image.

[0207] The first reference object and the second reference object are two different reference objects. The plurality of reference objects may include only the first reference object and the second reference object, or may include other reference objects besides the first reference object and the second reference object.

[0208] In this embodiment, a first reference speech is used to determine the lip movements of a first reference object to present the effect of the first reference object reading the first reference speech. By providing a first facial mask feature, it is possible to control that the first reference speech only applies to the facial area of ​​the first reference object and not to the facial areas of other reference objects besides the first reference object. By providing a second facial mask feature, it is possible to control that the second reference speech only applies to the facial area of ​​the second reference object and not to the facial areas of other reference objects besides the second reference object. Therefore, by providing different facial mask features, it is possible to accurately specify the reference object to be controlled.

[0209] 603. The computer device inputs a reference image, a noisy image, a first reference speech, and a first facial mask feature into a first video generation model to obtain a first video output by the first video generation model. The first video includes multiple reference objects, and the lip movements of the first reference objects are adapted to the first reference speech.

[0210] The computer device uses an image injection network to extract first image features from a reference image and noise features from a noisy image, then fuses the first image features and noise features to obtain second image features. A speech adaptation network extracts first speech features from a first reference speech, performs cross-attention processing on the second image features and the first speech features to obtain first cross features, and multiplies the first cross features by first facial mask features to obtain first fused features. A video generation network then generates a first video based on the first fused features.

[0211] The process of step 603 is the same as that of steps 503-505 above, and will not be repeated here.

[0212] 604. The computer device inputs a reference image, a noisy image, a second reference speech, and a second facial mask feature into a first video generation model to obtain a second video output by the first video generation model. The second video includes multiple reference objects, and the lip movements of the second reference objects are adapted to the second reference speech.

[0213] The computer device uses an image injection network to extract first image features from a reference image and noise features from a noisy image, then fuses the first image features and noise features to obtain second image features. A speech adaptation network extracts second speech features from a second reference speech, and cross-attention processing is applied to the second image features and second speech features to obtain second cross features. These second cross features are then multiplied by second facial mask features to obtain second fused features. Finally, a video generation network generates a second video based on these second fused features.

[0214] The process of step 604 is the same as that of steps 503-505 above, and will not be repeated here.

[0215] In one possible implementation, after generating the first and second videos, the computer device splices the first and second videos to obtain a third video. The third video includes multiple reference objects, and the lip movements of the first reference object among the multiple reference objects will change to present the effect of reading aloud. The lip movements of the second reference object among the multiple reference objects will also change to present the effect of reading aloud. This realizes the driving of the lip movements of different reference objects in the same video, which can be applied to scenarios that require driving multiple reference objects separately to generate a video.

[0216] Figure 7 This is a flowchart of another video generation method provided in the embodiments of this application, such as... Figure 7 As shown, reference image 701 includes a first reference object 711 and a second reference object 712. A first facial mask feature 703 is used to indicate the facial region of the first reference object 711 in reference image 701, and a second facial mask feature 705 is used to indicate the facial region of the second reference object 712 in reference image 701. The reference image 701, the noisy image 702, the first facial mask feature 703, and the first reference speech 704 are input into a first video generation model to obtain a first video 707. The first video 707 includes the first reference object 711 and the second reference object 712, and the lip movements of the first reference object 711 are adapted to the first reference speech 704. The reference image 701, the noisy image 702, the second facial mask feature 705, and the second reference speech 706 are input into the first video generation model to obtain a second video 708. The second video 708 includes the first reference object 711 and the second reference object 712, and the lip movements of the second reference object 712 are adapted to the second reference speech 706. The computer device splices the first video 707 and the second video 708 to obtain the third video 709. The third video 709 includes the first reference object 711 and the second reference object 712, and the effect is that the first reference object 711 reads the first reference speech 704 first, and then the second reference object 712 reads the first reference speech 706.

[0217] It should be noted that the embodiments in this application are only illustrated by taking the driving of lip shape changes of two reference objects as an example. In fact, based on the... Figure 6 The same method can be used in the embodiments to drive the lip movements of three or more reference objects to generate video. The number of reference objects is not limited in the embodiments of this application.

[0218] The method provided in this application embodiment can provide multiple facial mask features when the reference image includes multiple reference objects, each representing a different facial region of the reference object. Based on the different facial mask features, different reference voices can be used to guide the lip movements of different reference objects, thereby enabling voice-driven lip-syncing for multiple different reference objects and improving the diversity and flexibility of the video generation process.

[0219] The above embodiments illustrate the process of generating video based on reference images and reference speech. In addition to reference images and reference speech, reference text can also be introduced during the video generation process to specify the actions performed by the reference object. For details, please refer to the following. Figure 8 Examples of implementations. Figure 8 This is a flowchart of another video generation method provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 8 The method includes:

[0220] 801. The computer device acquires a first video generation model, which includes an image injection network, a speech adaptation network, a text processing network, and a video generation network.

[0221] The image injection network is connected to the speech adaptation network, and the output data of the image injection network is the input data of the speech-video network. The speech adaptation network and the text processing network are connected to the video generation network, and the output data of the speech adaptation network and the text processing network are the input data of the video generation network.

[0222] The network comprises three subnetworks: an image injection network for acquiring image features, a speech adaptation network for acquiring speech features and fusing these speech features with the features output by the image injection network, a text processing network for acquiring text features (specifically, text as input and text features as output), and a video generation network for generating videos based on the features output by the speech adaptation network and the text processing network. The speech adaptation network outputs fused features, and the text processing network outputs text features.

[0223] In one possible implementation, both the video generation network and the text processing network are obtained from a second video generation model, which is a trained model.

[0224] For example, this second video generation model is used to generate videos based on text.

[0225] 802. A computer device acquires a reference image, a noisy image, a reference speech, and a reference text. The reference image includes a reference object, the reference speech is the speech of the reference object, and the reference text is used to describe the actions performed by the reference object.

[0226] In one possible implementation, the computer device also acquires facial mask features, which are used to represent the facial region of a reference object in a reference image.

[0227] 803. The computer device extracts the first image features of the reference image and the noise features of the noisy image through the image injection network, and fuses the first image features and the noise features to obtain the second image features.

[0228] The process of step 803 is the same as that of step 503 above, and will not be repeated here.

[0229] 804. The computer device extracts the speech features of the reference speech through the speech adaptation network, and fuses the second image features and speech features to obtain the fused features.

[0230] In one possible implementation, the computer device extracts speech features from a reference speech via a speech adaptation network, performs cross-attention processing on the second image features and speech features to obtain cross features, multiplies the cross features with facial mask features to obtain intermediate features, and adds the intermediate features to the second image features to obtain fused features.

[0231] The process of step 804 is the same as that of step 504 above, and will not be repeated here.

[0232] 805. Computer equipment extracts text features from reference text through a text processing network. The reference text is used to describe the actions performed by the reference object.

[0233] The computer device inputs the reference text into the text processing network, which then extracts features from the reference text to obtain its text features.

[0234] Optionally, the text processing network can be a convolutional neural network, a recurrent neural network, a Transformer network, etc. The embodiments of this application do not limit the network structure of the text processing network.

[0235] 806. A computer device generates a video through a video generation network, based on fusion features and text features. The video includes a reference object, and the lip movements of the reference object are adapted to the reference speech. The actions of the reference object are the actions described in the reference text.

[0236] The computer device inputs fused features and text features into the video generation network, which then generates a video based on the fused features and text features.

[0237] In one possible implementation, the video generation network includes K diffusion transformation layers and a decoding layer. The diffusion transformation layers are used for denoising and can be dual-stream DiT, which processes both fused features and text features.

[0238] The method provided in this application embodiment includes a first video generation model comprising an image injection network, a speech adaptation network, and a video generation network. It provides the ability to process image and speech modal data and generate videos. The image injection network and speech adaptation network can transform images and speech into features easily understood by the video generation network. The image injection network acquires image features based on the image, and the speech adaptation network acquires speech features based on the speech. The image and speech features are then fused to obtain fused features. The video generation network then generates videos based on these fused features. This avoids the use of padding techniques to introduce images during the denoising stage, resulting in more natural, dynamic videos that are adapted to the reference speech, thus improving the video generation effect.

[0239] Furthermore, the video generation model also includes a text processing network, so reference text can be introduced to describe the actions performed by the reference object. The image injection network, speech adaptation network, and text processing network process the image, speech, and text respectively, transforming the image, speech, and text into features that are easy for the video generation network to understand. This enables video generation based on multimodal data such as images, speech, and text, further improving the controllability of the video generation process and enhancing the quality of video generation.

[0240] The training process of the first video generation model in the above embodiments can be found in the following: Figure 9 Examples of implementations. Figure 9 This is a flowchart illustrating a training method for a video generation model provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 9 The method includes:

[0241] 901. A computer device constructs a first video generation model based on an image injection network, a speech adaptation network, and a video generation network, with the image injection network, speech adaptation network, and video generation network connected sequentially.

[0242] Among them, the image injection network is used to obtain image features based on images, the speech adaptation network is used to obtain speech features based on speech and fuse the speech features with the features output by the image injection network, and the video generation network is used to generate video based on the features output by the speech adaptation network.

[0243] In one possible implementation, the computer device acquires a second video generation model, which is a trained model, and determines a video generation network within the second video generation model. The computer device constructs an image injection network and a speech adaptation network, connects the image injection network and the speech adaptation network, and connects the speech adaptation network and the video generation network to obtain a first video generation model.

[0244] In this model, the model parameters in the video generation network of the first video generation model are the same as those in the video generation network of the second video generation model. The model parameters in the image injection network and the speech adaptation network can be the initial model parameters.

[0245] In one possible implementation, the second video generation model includes a feature extraction layer. A computer device replicates the feature extraction layer in the second video generation model as a first feature extraction layer and a second feature extraction layer; based on the first feature extraction layer, the second feature extraction layer, and the feature fusion layer, an image injection network is constructed. Specifically, the first feature extraction layer is used to extract first sample image features from the sample reference image, and the second feature extraction layer is used to extract sample noise features from the sample noisy image.

[0246] The computer device constructs a feature fusion layer, and connects the first feature extraction layer and the second feature extraction layer to the feature fusion layer respectively to obtain an image injection network.

[0247] The model parameters of the first feature extraction layer of the image injection network are the same as those of the feature extraction layer of the second video generation model, and the model parameters of the second feature extraction layer of the image injection network are the same as those of the feature extraction layer of the second video generation model. The model parameters of the feature fusion layer can be the initialized model parameters.

[0248] In this implementation, the feature extraction layer in the image injection network can be directly copied from the trained feature extraction layer without rebuilding it. This is beneficial for improving the construction speed of the first video generation model and for improving the processing power and convergence efficiency of the image injection network.

[0249] 902. The computer device acquires a sample reference image, a sample noise image, a sample reference speech, and a labeled video. The sample reference image includes a sample reference object, the labeled video includes a sample reference object, and the lip movements of the sample reference object are adapted to the sample reference speech.

[0250] In one possible implementation, a computer device acquires a tagged video, which can be a real-life video recording, including a sample reference object speaking. The computer device extracts video frames from the tagged video and uses these frames as sample reference images, which include the sample reference object. The computer device then extracts speech from the tagged video and uses the extracted speech as sample reference speech. Therefore, this sample reference speech is the speech produced by the sample reference object speaking, and thus the lip movements of the sample reference object in the tagged video match this sample reference speech.

[0251] Optionally, the computer device acquires multiple raw videos and filters them to obtain labeled videos. The filtering process includes transition detection, face detection, text detection, and audio-visual synchronization detection. The filtered labeled videos meet the following criteria: no transitions, no text occlusion, presence of facial regions, and audio-visual synchronization. By filtering labeled videos, the quality of the labeled videos can be guaranteed, providing high-quality training data for model training.

[0252] Figure 10 This is a schematic diagram of a video filtering method provided in an embodiment of this application, such as... Figure 10 As shown, the original video is obtained. During the filtering stage, the original video is transcoded, for example, the frame rate of the transcoded video is 25 frames per second. Transition detection, face detection, text detection, and audio-visual synchronization detection are performed on the transcoded video to filter out videos with tags that have no transitions, no text occlusion, facial areas, and audio-visual synchronization.

[0253] 903. The computer device extracts the first sample image features of the sample reference image and the sample noise features of the sample noise image through the image injection network, and fuses the first sample image features and the sample noise features to obtain the second sample image features.

[0254] 904. The computer device extracts the sample speech features of the sample reference speech through the speech adaptation network, and fuses the second sample image features and the sample speech features to obtain the sample fusion features.

[0255] 905. Computer equipment generates sample videos based on sample fusion features through a video generation network.

[0256] The process of steps 903-905 is the same as that of steps 503-505 above, and will not be repeated here.

[0257] 906. The computer equipment trains the first video generation model based on sample videos and labeled videos.

[0258] The labeled videos are real videos, while the sample videos are intelligently generated by the first video generation model. The smaller the difference between the sample videos and the labeled videos generated by the first video generation model, the more accurate the model is. Therefore, the computer device adjusts the model parameters of the first video generation model based on the sample videos and labeled videos to obtain a more accurate model. The training objective is to reduce the difference between the sample videos and the labeled videos, thereby improving the video generation capability of the first video generation model.

[0259] In one possible implementation, the computer device determines a loss parameter based on sample videos and labeled videos, the loss parameter representing the degree of difference between the sample videos and labeled videos. The computer device then adjusts the model parameters of each network in the first video generation model with the goal of reducing this loss parameter during training.

[0260] The method provided in this application embodiment includes a first video generation model comprising an image injection network, a speech adaptation network, and a video generation network. It provides the ability to process image and speech modal data and generate videos. The image injection network and speech adaptation network can transform images and speech into features easily understood by the video generation network. The image injection network acquires image features based on the image, and the speech adaptation network acquires speech features based on the speech. The image and speech features are then fused to obtain fused features. The video generation network then generates videos based on these fused features. This avoids the use of padding techniques to introduce images during the denoising stage, resulting in more natural, dynamic videos that are adapted to the reference speech, thus improving the video generation effect.

[0261] In some embodiments, the second video generation model further includes a text processing network, and the first video generation model may also use the text processing network of the second video generation model. See below for details. Figure 11 Examples of implementations. Figure 11 This is a flowchart illustrating another training method for a video generation model provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 11 The method includes:

[0262] 1101. A computer device constructs a first video generation model based on an image injection network, a speech adaptation network, a text processing network, and a video generation network.

[0263] The image injection network is connected to the speech adaptation network, and the output data of the image injection network is the input data of the speech-video network. The speech adaptation network and the text processing network are connected to the video generation network, and the output data of the speech adaptation network and the text processing network are the input data of the video generation network.

[0264] Among them, the image injection network is used to obtain image features based on images, the speech adaptation network is used to obtain speech features based on speech and fuse the speech features with the features output by the image injection network, the text processing network is used to obtain text features based on text, and the video generation network is used to generate video using the features output by the speech adaptation network and the features output by the text processing network.

[0265] In one possible implementation, the computer device acquires a second video generation model, which is a trained model used to generate videos based on text. This second video generation model includes a text processing network and a video generation network. The computer device acquires the text processing network and video generation network of the second video generation model and constructs an image injection network and a speech adaptation network. The image injection network is connected to the speech adaptation network, the speech adaptation network is connected to the video generation network, and the text processing network and video generation network are connected to obtain a first video generation model.

[0266] In this model, the model parameters in the video generation network of the first video generation model are the same as those in the video generation network of the second video generation model. The model parameters in the text processing network of the first video generation model are also the same as those in the text processing network of the second video generation model. The model parameters in the image injection network and the speech adaptation network can be initialized model parameters.

[0267] 1102. The computer device acquires a sample reference image, a sample noise image, a sample reference speech, and a labeled video. The sample reference image includes a sample reference object, the labeled video includes a sample reference object, and the lip movements of the sample reference object are adapted to the sample reference speech.

[0268] 1103. The computer device extracts the first sample image features of the sample reference image and the sample noise features of the sample noise image through the image injection network, and fuses the first sample image features and the sample noise features to obtain the second sample image features.

[0269] 1104. The computer device extracts the sample speech features of the sample reference speech through the speech adaptation network, and fuses the second sample image features and the sample speech features to obtain the sample fusion features.

[0270] The process of steps 1102-1104 is the same as that of steps 902-904 above, and will not be repeated here.

[0271] 1105. Computer equipment extracts sample text features from sample reference text through a text processing network. The sample reference text is used to describe the actions performed by the reference object in the tagged video.

[0272] The process of step 1105 is the same as that of step 805 above, and will not be repeated here.

[0273] 1106. Computer equipment generates sample videos based on sample fusion features and sample text features through a video generation network.

[0274] The process of step 1106 is the same as that of step 806 above, and will not be repeated here.

[0275] 1107. The computer equipment trains the first video generation model based on sample videos and labeled videos.

[0276] The process of step 1107 is the same as that of step 906 above, and will not be repeated here.

[0277] Figure 12 This is a flowchart of another training method for a video generation model provided in an embodiment of this application, such as... Figure 12 As shown, the second video generation model includes a text processing network and a video generation network, and can only generate videos based on text. Based on the text processing and video generation networks, an image injection network and a speech adaptation network are added to extend image processing and speech processing capabilities, resulting in the first video generation model. The first video generation model can generate videos based on multimodal images, speech, and text. After constructing the first video generation model, it is trained using sample reference images, sample reference speech, sample facial mask features, sample reference text, and labeled videos. That is, the first video generation model generates sample videos based on sample reference images, sample reference speech, sample facial mask features, and sample reference text to reduce the difference between the sample videos and labeled videos, thereby adjusting the model parameters of each network in the first video generation model and training it.

[0278] In this embodiment, the video generation model also includes a text processing network. Therefore, reference text can be introduced to describe the actions performed by the reference object. The image injection network, speech adaptation network, and text processing network process the image, speech, and text respectively, and convert the image, speech, and text into features that are easy for the video generation network to understand. This enables video generation based on multimodal data such as images, speech, and text, further improving the controllability of the video generation process and the effect of video generation.

[0279] Furthermore, since the text processing network is obtained from the second video generation model, the processing capabilities of the second video generation model can be directly utilized, which is beneficial for improving the training speed and text processing capabilities of the first video generation model.

[0280] With the development of artificial intelligence technology, it is possible to intelligently generate videos of virtual characters, which are digital humans created based on digital technology. Therefore, the method provided in this application embodiment can generate virtual character videos based on voice-driven processes. See below for details. Figure 13 Examples of implementations. Figure 13 This is a flowchart illustrating a voice-driven virtual character video generation method provided in an embodiment of this application. The method is executed by a computer device, such as... Figure 13 As shown, the method includes the following steps.

[0281] 1301. The computer device acquires a first video generation model, which includes an image injection network, a speech adaptation network, and a video generation network connected in sequence.

[0282] 1302. A computer device acquires a virtual character image, a character's voice, and facial mask features, wherein the virtual character image includes a virtual character, the character's voice is the voice to be read aloud by the virtual character, and the facial mask features are used to indicate the facial region of the virtual character in the virtual character image.

[0283] 1303. The computer device extracts the first image features of the virtual character image and the noise features of the noisy image through the image injection network, and fuses the first image features and the noise features to obtain the second image features, wherein the reference image includes the virtual character.

[0284] 1304. The computer device extracts the speech features of a person's speech through a speech adaptation network, and fuses the second image features and speech features to obtain fused features.

[0285] 1305. A computer device generates a first virtual character video through a video generation network based on fusion features. The first virtual character video includes a virtual character, and the lip movements of the virtual character are the same as those of the character when the speech is read aloud.

[0286] 1306. The computer device uses human voice to dub the first virtual character video to obtain the second virtual character video.

[0287] The above-mentioned voice-driven virtual character video generation method can be applied to any scenario. The following examples illustrate its application to scenarios such as virtual anchors, virtual meetings, and intelligent customer service assistants.

[0288] 1. Generating Virtual Anchor Scenes. In fields such as live streaming, entertainment, and education, providing photos of virtual anchors and combining them with voice input, the method described in this application can intelligently generate virtual anchor videos with natural movements and synchronized facial and lip movements with voice. The virtual anchor can generate dynamic performances that match the rhythm of the voice provided by the user, greatly enhancing the realism and immersion of virtual interaction.

[0289] 2. Voice-Driven Virtual Meetings. In remote work, video conferencing, and social platforms, the method described in this application integrates voice-driven facial lip movements and text-driven body language into the performance of virtual participants, enhancing the naturalness of remote interaction and improving the emotional expression and communication efficiency of participants in virtual meetings. The virtual participant can be a virtual image generated based on a real participant, or a virtual image designed or selected by a real participant.

[0290] 3. Intelligent Customer Service Assistant. In the intelligent customer service assistant, corresponding reply text is generated based on the user's question. The reply text is then converted into speech, and using the method provided in this application embodiment, the speech is used to drive the virtual customer service representative's facial lip movements, generating a virtual customer service representative's voice reply video. This enhances the realism and emotional expression of human-computer interaction in the intelligent customer service assistant, making the virtual customer service representative's responses more vivid and friendly, thereby improving the user experience.

[0291] Figure 14 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application. See also... Figure 14 The device includes:

[0292] The model acquisition module 1401 is used to acquire a first video generation model, which includes an image injection network, a speech adaptation network and a video generation network connected in sequence.

[0293] Image processing module 1402 is used to extract first image features of a reference image and noise features of a noisy image through an image injection network, and fuse the first image features and noise features to obtain second image features, wherein the reference image includes a reference object;

[0294] The speech processing module 1403 is used to extract speech features of a reference speech through a speech adaptation network, fuse the second image features and speech features to obtain fused features, where the reference speech is the speech of the reference object;

[0295] The video generation module 1404 is used to generate a video based on fusion features through a video generation network. The video includes a reference object, and the lip movements of the reference object are adapted to the reference speech.

[0296] The video generation apparatus provided in this application embodiment includes a first video generation model comprising an image injection network, a speech adaptation network, and a video generation network. It provides the ability to process image and speech modal data and generate videos. The image injection network and speech adaptation network can transform images and speech into features easily understood by the video generation network. The image injection network acquires image features based on the image, and the speech adaptation network acquires speech features based on the speech. The image and speech features are then fused to obtain fused features. The video generation network then generates videos based on these fused features. This avoids the use of padding techniques to introduce images during the denoising stage, resulting in more natural, dynamic videos that are adapted to the reference speech, thus improving the video generation effect.

[0297] Optionally, see Figure 15 The reference image has M frames, and the noisy image has M frames, where M is an integer greater than 1. The image injection network includes a first feature extraction layer, a second feature extraction layer, and a feature fusion layer; the image processing module 1402 is used for:

[0298] The first feature extraction layer performs 3D convolution on the M reference images to obtain the features of the first N images, where N is a positive integer less than M.

[0299] The second feature extraction layer performs three-dimensional convolution on the M-frame noisy images to obtain the noise features of N frames.

[0300] The feature fusion layer fuses the first image features of N frames with the noise features of N frames to obtain the second image features.

[0301] Optionally, see Figure 15 The image injection network includes a first feature fusion layer and a second feature fusion layer. The image processing module 1402 is used for:

[0302] The first feature fusion layer adds the first image features of N frames to the noise features of N frames to obtain intermediate image features, and the number of frames of the intermediate image features is N frames.

[0303] The second feature fusion layer concatenates the first image features of the first frame with the intermediate image features in the first image features of N frames to obtain the second image features. The number of frames for the second image features is N+1.

[0304] Optionally, see Figure 15 The device also includes a mask feature acquisition module 1405, which is used to acquire facial mask features, which are used to represent the facial region of the reference object in the reference image;

[0305] Voice processing module 1403 is used for:

[0306] Cross-attention processing is applied to the second image features and speech features to obtain cross features;

[0307] The intermediate feature is obtained by multiplying the cross feature with the facial mask feature;

[0308] The second image feature and the intermediate feature are added together to obtain the fused feature.

[0309] Optionally, see Figure 15 The reference image includes multiple reference objects, the reference speech includes a first reference speech and a second reference speech, the facial mask features include a first facial mask feature and a second facial mask feature, the first facial mask feature is used to represent the facial region of the first reference object in the reference image, the second facial mask feature is used to represent the facial region of the second reference object in the reference image, the fusion features include a first fusion feature and a second fusion feature, the first fusion feature is obtained based on the second image features, the speech features of the first reference speech and the first mask feature, the second fusion feature is obtained based on the second image features, the speech features of the second reference speech and the second mask feature; the video generation module 1404 is used for:

[0310] A first video is generated using a video generation network based on a first fusion feature. The first video includes multiple reference objects, and the lip movements of the first reference objects are adapted to the first reference speech.

[0311] A second video is generated using a video generation network based on a second fusion feature. The second video includes multiple reference objects, and the lip movements of the second reference objects are adapted to the second reference speech.

[0312] Optionally, see Figure 15 The speech adaptation network includes K third feature fusion layers, and the video generation network includes K diffusion transformation layers, where K is an integer greater than 1; the speech processing module 1403 is used for:

[0313] The first third feature fusion layer fuses the second image features and the speech features to obtain the first third fusion feature;

[0314] The i-th third feature fusion layer fuses the (i-1)-th third fusion feature and the (i-1)-th fourth fusion feature to obtain the i-th third fusion feature; where i is a positive integer not greater than K, and the i-th fourth fusion feature is obtained through the i-th diffusion transformation layer.

[0315] Optionally, see Figure 15 The voice processing module 1403 is used for:

[0316] Cross-attention processing is applied to the second image feature and the speech feature to obtain the first cross feature;

[0317] The first cross feature is multiplied by the face mask feature to obtain the intermediate feature, which is used to represent the facial region of the reference object in the reference image.

[0318] The second image feature and the intermediate feature are added together to obtain the first third fusion feature.

[0319] Optionally, see Figure 15 The voice processing module 1403 is used for:

[0320] Cross-attention processing is performed on the (i-1)th third fusion feature and the (i-1)th fourth fusion feature to obtain the i-th cross feature;

[0321] Add the i-th cross feature and the (i-1)-th fourth fusion feature to obtain the i-th third fusion feature.

[0322] Optionally, see Figure 15 The video generation network also includes a decoding layer; the video generation module 1404 is used for:

[0323] The first diffusion transformation layer is used to denoise the first third fusion feature to obtain the first fourth fusion feature;

[0324] The ith third fusion feature is denoised through the ith diffusion transformation layer to obtain the ith fourth fusion feature;

[0325] The video is obtained by decoding the Kth fourth fusion feature through the decoding layer.

[0326] Optionally, see Figure 15 The first video generation model also includes a text processing network, which is connected to the video generation network;

[0327] The device also includes a text processing module 1406, which is used to extract text features of a reference text through a text processing network. The reference text is used to describe the actions performed by the reference object.

[0328] The video generation module 1404 is used to generate a video through a video generation network based on fused features and text features. The video includes a reference object, and the lip movements of the reference object are adapted to the reference speech. The actions of the reference object are the actions described in the reference text.

[0329] It should be noted that the video generation apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video generation apparatus and the video generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0330] Figure 16 This is a schematic diagram of the structure of a training device for a video generation model provided in an embodiment of this application. See also... Figure 16 The device includes:

[0331] The model building module 1601 is used to build a first video generation model based on an image injection network, a speech adaptation network, and a video generation network, with the image injection network, speech adaptation network, and video generation network connected in sequence.

[0332] The sample data acquisition module 1602 is used to acquire sample reference images, sample noise images, sample reference speech and labeled videos. The sample reference images include sample reference objects, the labeled videos include sample reference objects, and the lip movements of the sample reference objects are adapted to the sample reference speech.

[0333] Image processing module 1603 is used to extract first sample image features of sample reference image and sample noise features of sample noise image through image injection network, and fuse first sample image features and sample noise features to obtain second sample image features;

[0334] The speech processing module 1604 is used to extract sample speech features of sample reference speech through a speech adaptation network, and fuse the second sample image features and sample speech features to obtain sample fusion features.

[0335] The video generation module 1605 is used to generate sample videos based on sample fusion features through a video generation network.

[0336] Model training module 1606 is used to train the first video generation model based on sample videos and labeled videos.

[0337] The training apparatus for the video generation model provided in this application embodiment includes an image injection network, a speech adaptation network, and a video generation network. It provides the ability to process image and speech modal data and generate videos. The image injection network and speech adaptation network can transform images and speech into features easily understood by the video generation network. The image injection network acquires image features based on the image, and the speech adaptation network acquires speech features based on the speech. The image and speech features are then fused to obtain fused features. The video generation network then generates videos based on these fused features. This avoids the use of padding techniques to introduce images during the denoising stage, resulting in more natural, dynamic videos that are adapted to the reference speech, thus improving the video generation effect.

[0338] Optionally, the model building module 1601 is also used for:

[0339] The trained feature extraction layer is copied into a first feature extraction layer and a second feature extraction layer;

[0340] An image injection network is constructed based on the first feature extraction layer, the second feature extraction layer, and the feature fusion layer.

[0341] The first feature extraction layer is used to extract the first sample image features of the sample reference image, and the second feature extraction layer is used to extract the sample noise features of the sample noise image.

[0342] Optionally, the first video generation model further includes a text processing network connected to the video generation network; the device further includes a text processing module for extracting sample text features of sample reference text through the text processing network, the sample reference text being used to describe the actions performed by the reference object in the labeled video;

[0343] The video generation module 1605 is used to generate sample videos based on sample fusion features and sample text features through a video generation network.

[0344] It should be noted that the video generation apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the training device for the video generation model and the training method for the video generation model provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0345] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to implement the operations performed in the video generation method of the above embodiments, or to implement the operations performed in the training method of the video generation model of the above embodiments.

[0346] Optionally, the computer device is provided as a terminal. Figure 17 A schematic diagram of the structure of a terminal 1700 provided in an exemplary embodiment of this application is shown.

[0347] Terminal 1700 includes a processor 1701 and a memory 1702.

[0348] Processor 1701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0349] Memory 1702 may include one or more computer-readable storage media, which may be non-transitory. Memory 1702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 1702 are used to store at least one computer program, which is used by processor 1701 to implement the video generation method or video generation model training method provided in the method embodiments of this application.

[0350] In some embodiments, the terminal 1700 may also optionally include: a peripheral device interface 1703 and at least one peripheral device. The processor 1701, memory 1702, and peripheral device interface 1703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1703 via a bus, signal line, or circuit board. Optionally, the peripheral device includes at least one of: a radio frequency circuit 1704, a display screen 1705, a camera assembly 1706, an audio circuit 1707, and a power supply 1708.

[0351] Peripheral interface 1703 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1701 and memory 1702. In some embodiments, processor 1701, memory 1702 and peripheral interface 1703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1701, memory 1702 and peripheral interface 1703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0352] The radio frequency (RF) circuit 1704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1704 can communicate with other devices through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1704 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0353] Display screen 1705 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1701 for processing. In this case, display screen 1705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1705, disposed on the front panel of terminal 1700; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1700 or in a folded design; in still other embodiments, display screen 1705 may be a flexible display screen, disposed on a curved or folded surface of terminal 1700. Furthermore, display screen 1705 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1705 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0354] The camera assembly 1706 is used to acquire images or videos. Optionally, the camera assembly 1706 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal 1700, and the rear-facing camera is disposed on the back of the terminal 1700. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1706 may also include a flash. The flash may be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0355] The audio circuit 1707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1701 for processing, or input to the radio frequency circuit 1704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1701 or the radio frequency circuit 1704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1707 may also include a headphone jack.

[0356] Power supply 1708 is used to power the various components in terminal 1700. Power supply 1708 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1708 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0357] Those skilled in the art will understand that Figure 17 The structure shown does not constitute a limitation on terminal 1700 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0358] Optionally, the computer device is provided as a server. Figure 18 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1800 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1801 and one or more memories 1802. The memories 1802 store at least one computer program, which is loaded and executed by the processor 1801 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0359] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the video generation method of the above embodiments, or to implement the operations performed by the training method of the video generation model of the above embodiments.

[0360] This application also provides a computer program product, including a computer program loaded and executed by a processor to perform the operations performed by the video generation method of the above embodiments, or to perform the operations performed by the training method of the video generation model of the above embodiments.

[0361] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0362] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A method of video generation, the method comprising: The method includes: Obtain a first video generation model, which includes an image injection network, a speech adaptation network, and a video generation network connected in sequence. The image injection network extracts first image features from a reference image and noise features from a noisy image, and fuses the first image features and the noise features to obtain second image features. The reference image includes a reference object. The speech features of the reference speech are extracted through the speech adaptation network, and the second image features and the speech features are fused to obtain fused features. The reference speech is the speech of the reference object. The video is generated by the video generation network based on the fusion features. The video includes the reference object, and the lip movements of the reference object are adapted to the reference speech.

2. The method of claim 1, wherein, The reference image has M frames, the noisy image has M frames, where M is an integer greater than 1, and the image injection network includes a first feature extraction layer, a second feature extraction layer, and a feature fusion layer. The process involves extracting first image features from a reference image and noise features from a noisy image through the image injection network, then fusing the first image features and the noise features to obtain second image features, including: The first feature extraction layer performs three-dimensional convolution on the M reference images to obtain the features of the first N images, where N is a positive integer less than M. The second feature extraction layer performs three-dimensional convolution on the M-frame noise images to obtain N-frame noise features. The feature fusion layer fuses the first image features of the N frames with the noise features of the N frames to obtain the second image features.

3. The method of claim 2, wherein, The image injection network includes a first feature fusion layer and a second feature fusion layer. The second image feature is obtained by fusing the first image features of N frames with the noise features of N frames through the feature fusion layer, including: The first feature fusion layer adds the first image features of the N frames to the noise features of the N frames to obtain intermediate image features, wherein the number of frames of the intermediate image features is N frames. The second feature fusion layer concatenates the first image feature of the first frame in the N frames of the first image feature with the intermediate image features to obtain the second image feature, and the number of frames of the second image feature is N+1 frames.

4. The method of claim 1, wherein, The method further includes: Obtain facial mask features, which are used to represent the facial region of the reference object in the reference image; The fusion of the second image features and the speech features to obtain fused features includes: Cross-attention processing is performed on the second image features and the speech features to obtain cross features; The intermediate feature is obtained by multiplying the intersection feature with the facial mask feature; The second image feature and the intermediate feature are added together to obtain the fused feature.

5. The method of claim 4, wherein, The reference image includes multiple reference objects, the reference speech includes a first reference speech and a second reference speech, the facial mask feature includes a first facial mask feature and a second facial mask feature, the first facial mask feature is used to represent the facial region of the first reference object in the reference image, the second facial mask feature is used to represent the facial region of the second reference object in the reference image, and the fusion feature includes a first fusion feature and a second fusion feature, the first fusion feature is obtained based on the second image feature, the speech feature of the first reference speech and the first mask feature, and the second fusion feature is obtained based on the second image feature, the speech feature of the second reference speech and the second mask feature; The process of generating a video through the video generation network based on the fusion features includes: A first video is generated through the video generation network based on the first fusion feature. The first video includes the plurality of reference objects, and the lip movements of the first reference objects are adapted to the first reference speech. A second video is generated using the video generation network based on the second fusion feature. The second video includes the plurality of reference objects, and the lip movements of the second reference objects are adapted to the second reference speech.

6. The method according to claim 1, characterized in that, The speech adaptation network includes K third feature fusion layers, and the video generation network includes K diffusion transformation layers, where K is an integer greater than 1; the fusion of the second image features and the speech features to obtain fused features includes: The first third feature fusion layer fuses the second image features and the speech features to obtain the first third fusion feature; The i-th third feature is obtained by fusing the (i-1)-th third feature and the (i-1)-th fourth feature through the i-th third feature fusion layer; where i is a positive integer not greater than K, and the i-th fourth feature is obtained through the i-th diffusion transformation layer.

7. The method according to claim 6, characterized in that, The first third fused feature is obtained by fusing the second image feature and the speech feature, including: The second image feature and the speech feature are subjected to cross-attention processing to obtain the first cross feature; The first cross feature is multiplied by the facial mask feature to obtain the intermediate feature, which is used to represent the facial region of the reference object in the reference image; The second image feature and the intermediate feature are added together to obtain the first third fusion feature.

8. The method according to claim 6, characterized in that, The fusion of the (i-1)th third fusion feature and the (i-1)th fourth fusion feature yields the i-th third fusion feature, including: Cross-attention processing is performed on the (i-1)th third fusion feature and the (i-1)th fourth fusion feature to obtain the i-th cross feature; The i-th cross feature and the (i-1)-th fourth fusion feature are added together to obtain the i-th third fusion feature.

9. The method according to claim 6, characterized in that, The video generation network further includes a decoding layer; the process of generating a video based on the fusion features using the video generation network includes: The first diffusion transformation layer is used to denoise the first third fusion feature to obtain the first fourth fusion feature; The ith third fusion feature is denoised through the ith diffusion transformation layer to obtain the ith fourth fusion feature; The video is obtained by decoding the Kth fourth fusion feature through the decoding layer.

10. The method according to any one of claims 1 to 9, characterized in that, The first video generation model further includes a text processing network, which is connected to the video generation network; the method further includes: The text processing network extracts text features from a reference text, which is used to describe the actions performed by the reference object. The process of generating a video through the video generation network based on the fusion features includes: The video is generated by the video generation network based on the fusion features and the text features. The video includes a reference object whose lip movements are adapted to the reference speech, and whose actions are described in the reference text.

11. A training method for a video generation model, characterized in that, The method includes: A first video generation model is constructed based on an image injection network, a speech adaptation network, and a video generation network, wherein the image injection network, the speech adaptation network, and the video generation network are connected in sequence. Acquire a sample reference image, a sample noise image, a sample reference speech, and a labeled video. The sample reference image includes a sample reference object, the labeled video includes the sample reference object, and the lip movements of the sample reference object are adapted to the sample reference speech. The image injection network extracts the first sample image features of the sample reference image and the sample noise features of the sample noise image, and then fuses the first sample image features and the sample noise features to obtain the second sample image features. The speech features of the sample reference speech are extracted through the speech adaptation network, and the sample image features and the sample speech features are fused to obtain the sample fusion features. Sample videos are generated using the video generation network based on the sample fusion features; The first video generation model is trained based on the sample video and the labeled video.

12. The method according to claim 11, characterized in that, The method further includes: The trained feature extraction layer is copied into a first feature extraction layer and a second feature extraction layer; The image injection network is constructed based on the first feature extraction layer, the second feature extraction layer, and the feature fusion layer; The first feature extraction layer is used to extract the first sample image features of the sample reference image, and the second feature extraction layer is used to extract the sample noise features of the sample noise image.

13. The method according to claim 11, characterized in that, The first video generation model further includes a text processing network, which is connected to the video generation network; the method further includes: The text processing network extracts sample text features from the sample reference text, which is used to describe the actions performed by the reference object in the tagged video. The step of generating sample videos through the video generation network based on the sample fusion features includes: The sample video is generated using the video generation network based on the sample fusion features and the sample text features.

14. A video generation apparatus, characterized in that, The device includes: The model acquisition module is used to acquire a first video generation model, which includes an image injection network, a speech adaptation network, and a video generation network connected in sequence. The image processing module is used to extract first image features of a reference image and noise features of a noisy image through the image injection network, and fuse the first image features and the noise features to obtain second image features, wherein the reference image includes a reference object; The speech processing module is used to extract speech features of a reference speech through the speech adaptation network, fuse the second image features and the speech features to obtain fused features, wherein the reference speech is the speech of the reference object; A video generation module is used to generate a video based on the fusion features through the video generation network, wherein the video includes the reference object and the lip movements of the reference object are adapted to the reference speech.

15. A training device for a video generation model, characterized in that, The device includes: The model building module is used to build a first video generation model based on an image injection network, a speech adaptation network, and a video generation network, wherein the image injection network, the speech adaptation network, and the video generation network are connected in sequence. The sample data acquisition module is used to acquire sample reference images, sample noise images, sample reference speech, and labeled videos. The sample reference images include sample reference objects, the labeled videos include the sample reference objects, and the lip movements of the sample reference objects are adapted to the sample reference speech. The image processing module is used to extract the first sample image features of the sample reference image and the sample noise features of the sample noise image through the image injection network, and fuse the first sample image features and the sample noise features to obtain the second sample image features; The speech processing module is used to extract sample speech features of the sample reference speech through the speech adaptation network, and fuse the second sample image features and the sample speech features to obtain sample fusion features; A video generation module is used to generate sample videos based on the sample fusion features through the video generation network; The model training module is used to train the first video generation model based on the sample video and the labeled video.

16. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the video generation method as described in any one of claims 1 to 10, or to perform the operations performed by the training method for the video generation model as described in any one of claims 11 to 13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the operations performed by the video generation method as described in any one of claims 1 to 10, or to implement the operations performed by the training method for the video generation model as described in any one of claims 11 to 13.

18. A computer program product, comprising a computer program, characterized in that, The computer program is loaded and executed by a processor to perform the operations performed by the video generation method as described in any one of claims 1 to 10, or to perform the operations performed by the training method for the video generation model as described in any one of claims 11 to 13.