Video generation method and apparatus, electronic device, and storage medium

Abstract

This application discloses a video generation method, apparatus, electronic device, and storage medium, relating to the field of artificial intelligence technology, particularly computer vision, deep learning, and large model technologies, and applicable to scenarios such as AIGC, digital humans, and intelligent e-commerce. The specific implementation scheme is as follows: Obtain a reference human body signal corresponding to a reference image displaying a target object and a driving human body signal corresponding to a driving video; perform dynamic feature extraction on the reference human body signal and the driving human body signal respectively to obtain a first dynamic feature and a second dynamic feature; extract texture features from the reference image to obtain a first texture feature; perform cross-attention calculation on the first dynamic feature, the second dynamic feature, and the first texture feature to obtain a second texture feature; concatenate the second texture feature, the second dynamic feature, and noise to obtain a first concatenated feature; perform denoising processing based on the first concatenated feature to generate a target video displaying the target object.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large models, and can be applied to AIGC (Artificial Intelligence Generated Content), digital humans, intelligent e-commerce, and other scenarios. Specifically, it relates to a video generation method, device, electronic device, and storage medium. Background Technology

[0002] Human body motion technology refers to the technology of generating a video of the limb movements of a target object based on any video of human movement. The target object in the generated human body motion video has the same limb movements, gestures, facial expressions and lip movements as the person who is being motioned. Summary of the Invention

[0003] This application provides a video generation method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this application, a video generation method is provided, comprising:

[0005] Acquire a reference human body signal corresponding to a reference image displaying a target object and a driving human body signal corresponding to a driving video; wherein, the reference human body signal is used to indicate the human posture of the target object, and the driving human body signal is used to indicate the human posture of the driven object in the driving video;

[0006] Dynamic features are extracted from the reference human body signal and the driving human body signal respectively to obtain the first dynamic feature and the second dynamic feature;

[0007] The reference image is subjected to texture feature extraction to obtain the first texture feature;

[0008] Cross-attention calculation is performed on the first dynamic feature, the second dynamic feature, and the first texture feature to obtain the second texture feature;

[0009] The second texture feature, the second dynamic feature, and the noise are spliced ​​together to obtain the first spliced ​​feature;

[0010] Denoising is performed based on the first splicing feature to generate a target video displaying the target object.

[0011] According to another aspect of this application, a video generation apparatus is provided, comprising:

[0012] The first acquisition module is used to acquire a reference human body signal corresponding to a reference image displaying a target object and a driving human body signal corresponding to a driving video; wherein, the reference human body signal is used to indicate the human body posture of the target object, and the driving human body signal is used to indicate the human body posture of the driving object in the driving video;

[0013] The dynamic feature extraction module is used to extract dynamic features from the reference human body signal and the driving human body signal respectively to obtain the first dynamic feature and the second dynamic feature.

[0014] The texture feature extraction module is used to extract texture features from the reference image to obtain the first texture feature;

[0015] The second acquisition module is used to perform cross-attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain the second texture feature;

[0016] The feature splicing module is used to splice the second texture feature, the second dynamic feature, and noise to obtain the first spliced ​​feature;

[0017] The generation module is used to perform noise reduction processing based on the first splicing features to generate a target video displaying the target object.

[0018] According to another aspect of this application, an electronic device is provided, comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in the above embodiments.

[0022] According to another aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method described in the above embodiments.

[0023] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the above embodiments.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0025] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:

[0026] Figure 1 A schematic flowchart illustrating a video generation method provided in an embodiment of this application;

[0027] Figure 2 A flowchart illustrating a video generation method provided in another embodiment of this application;

[0028] Figure 3 A flowchart illustrating a video generation method provided in another embodiment of this application;

[0029] Figure 4 A schematic diagram of a video generation process provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the structure of a video generation apparatus provided in an embodiment of this application;

[0031] Figure 6 This is a block diagram of an electronic device used to implement the video generation method of the embodiments of this application. Detailed Implementation

[0032] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0033] The video generation method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings.

[0034] Figure 1 This is a schematic flowchart of a video generation method provided in an embodiment of this application.

[0035] The video generation method of this application embodiment can be executed by the video generation device of this application embodiment, which can be configured in an electronic device.

[0036] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0037] like Figure 1 As shown, the video generation method includes:

[0038] Step 101: Obtain the reference human body signal corresponding to the reference image displaying the target object and the driving human body signal corresponding to the driving video.

[0039] The reference human body signal can be used to indicate the human posture of the target object in the reference image, and the driving human body signal can be used to indicate the human posture of the driven object in the driving video. Furthermore, the target object and the driving object can be the same or different; this application does not impose any limitations on this.

[0040] For example, key point detection can be performed on the reference image to obtain human key point information of the target object. Based on the human key point information of the target object, a pose heatmap of the target object can be obtained, and the pose heatmap of the target object can be used as a reference human signal.

[0041] For example, if the reference image is a frame extracted from the driving video, then the target object is the same as the driving object. Key point detection can be performed on each frame of the driving video. Based on the human key point information of the driving object in each frame, the pose heatmap corresponding to the reference in each frame can be obtained, and the pose heatmap of the driving object in each frame can be used as the driving human signal.

[0042] Since hand movements may be obscured by fingers, and lip movements are more frequent, for example, coarse-grained movements such as limbs and torso can be represented by sparser 2D keypoints, while fine-grained movements such as face and hand can be represented by more accurate 3D face and hand reconstruction renderings. In other words, 2D keypoints of the human torso and limbs, and 3D renderings of hands and face can be used as driving signals for the human body.

[0043] Compared to 2D keypoints, 3D rendering maps can better represent motion postures and occlusion relationships, and have a more accurate ability to express complex gestures and frequently moving lip movements. Therefore, using 3D rendering maps to represent fine-grained areas such as hands and faces can improve the accuracy of driving human body signals. This allows for video generation based on driving human body signals, thus improving the accuracy of human movements in the generated video. Conversely, representing coarse-grained areas such as limbs and torso using 2D keypoints can reduce computational load and improve video generation efficiency while ensuring the accuracy of torso and limb movements.

[0044] Step 102: Dynamic feature extraction is performed on the reference human body signal and the driving human body signal respectively to obtain the first dynamic feature and the second dynamic feature.

[0045] For example, a dynamic feature encoder can be used to extract dynamic features from a reference human body signal to obtain a first dynamic feature, and a dynamic feature encoder can be used to extract dynamic features from a driving human body signal to obtain a second dynamic feature.

[0046] For example, the dimensions of the first dynamic feature and the second dynamic feature can be H*W*C, where H, W, and C represent the height, width, and number of channels, respectively.

[0047] Step 103: Extract texture features from the reference image to obtain the first texture feature.

[0048] A Variational Autoencoder (VAE) is a generative model that combines the ideas of an autoencoder and variational Bayesian inference. A VAE learns a latent representation of the input data and can generate new, similar data samples based on that representation. A VAE consists of an encoder and a decoder. The encoder maps the input data to points in a latent space, i.e., the latent representation. The decoder then generates new, similar data samples based on this latent representation.

[0049] For example, a reference image can be input into a VAE encoder, and the VAE encoder can be used to extract texture features from the reference image to obtain the first texture feature.

[0050] Step 104: Perform cross-attention calculation on the first dynamic feature, the second dynamic feature, and the first texture feature to obtain the second texture feature.

[0051] For example, for each frame of an image in a driving video, the second dynamic feature can be used as a query vector, the first dynamic feature as a key vector, and the first texture vector as a value vector. The second texture feature is obtained by cross-attention calculation.

[0052] Since the human pose information corresponding to the driving video is usually different from the human pose information of the reference image, the misalignment between texture and structural information may lead to inaccurate texture in the generated image. In this application, texture features that are aligned with the motion structure can be obtained through an attention mechanism.

[0053] Step 105: The second texture feature, the second dynamic feature, and the noise are spliced ​​together to obtain the first spliced ​​feature.

[0054] For example, the noise used can be Gaussian noise. For each frame of the driving video, the second texture feature, the second dynamic feature and the noise can be spliced ​​together to obtain the first spliced ​​feature corresponding to each frame.

[0055] Step 106: Perform noise reduction processing based on the first splicing feature to generate a target video displaying the target object.

[0056] For example, the first stitched features corresponding to each frame of the driving video can be input into a denoising network. The denoising network progressively removes noise, and then a VAE decoder is used to decode the features output by the denoising network to generate a target video displaying the target object. The target video displays the target object, and the target object's actions are consistent with the actions of the driving object in the driving video. Thus, it is possible to generate a video of the target object's actions based on a given driving video that drives human actions.

[0057] For example, the denoising network can be a U-Net network, which may include convolutional layers, max pooling layers (downsampling), attention layers, deconvolutional layers (upsampling), and ReLU non-linear activation functions.

[0058] In this embodiment, by performing dynamic feature encoding on the reference human body signal corresponding to the reference image and the driving human body signal corresponding to the driving video, a first dynamic feature and a second dynamic feature are obtained. Texture features are extracted from the reference image to obtain a first texture feature. Cross-attention calculation is performed on the first dynamic feature, the second dynamic feature, and the first texture feature. Through the attention mechanism, a second texture feature that aligns the texture with the motion structure is obtained, thus achieving effective alignment between the driving human body signal of the driving object and the reference image. Based on the second texture feature and the second dynamic feature, the target video is generated, improving the accuracy of the texture in the generated video and improving the quality of the generated video.

[0059] Figure 2 This is a flowchart illustrating a video generation method provided in another embodiment of this application.

[0060] like Figure 2 As shown, the video generation method includes:

[0061] Step 201: Obtain the reference human body signal corresponding to the reference image displaying the target object and the driving human body signal corresponding to the driving video.

[0062] In this application, step 201 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0063] For example, if the target object and the driving object are the same, keypoint detection can be performed on each frame of the driving video to obtain the human keypoints of the driving object. Then, based on each frame of the driving video, 3D face modeling and 3D hand modeling can be performed to obtain the first 3D face model and 3D hand model of the driving object. The first 3D face model and 3D hand model are then rendered to obtain the first 3D face rendering image and the 3D hand rendering image. Finally, based on the keypoints of the torso and limbs in the human keypoints, the first 3D face rendering image, and the 3D hand rendering image, the driving human body signal is obtained. The driving human body signal can include the keypoints of the torso and limbs of the driving object, the first 3D face rendering image, and the 3D hand rendering image. Here, the human body keypoints can be 2D keypoints.

[0064] For example, the driving video is a video of the upper body movement of the driving object. The key points of the torso and upper limbs of the driving object, the first 3D face rendering, and the 3D hand rendering can be used as driving human body signals.

[0065] For example, during the training phase, a frame can be extracted from the driving video as a reference image. At this time, the target object is the same as the driving object, and the key points of the torso and upper limbs, the 3D face rendering, and the 3D hand rendering of the driving object can be used as driving human body signals.

[0066] Because 3D rendered images can better represent motion postures and occlusion relationships, they have a more accurate ability to express complex gestures and frequently moving lip movements. Therefore, using the 3D face and hand rendered images of the driving object as driving body signals to generate videos that drive the target object can improve the accuracy of the target object's human body movements in the generated video. In addition, using key points of the human torso and limbs as driving body signals can reduce the amount of computation and improve video generation efficiency while ensuring the accuracy of torso and limb movements.

[0067] For example, if the target object is different from the driving object, the first face shape coefficient of the target object can be obtained, and the second face shape coefficient of the driving object in the first three-dimensional face model can be replaced with the first face shape coefficient to obtain the second three-dimensional face model. The second three-dimensional face model is then rendered to obtain the second three-dimensional face rendering image. Then, the driving human body signal is obtained based on the key points of the torso and limbs of the driving object, the three-dimensional hand rendering image, and the second three-dimensional face rendering image.

[0068] The three-dimensional face model can include head pose coefficients, face shape coefficients, and face expression coefficients. The face shape coefficients refer to the feature coefficients used to describe the shape of the face in the three-dimensional face model. The face shape coefficients can be used to describe the overall shape and structure of the face.

[0069] For example, during the reasoning phase, the target object is different from the driving object. The key points of the torso and limbs of the driving object, the 3D hand rendering, and the 3D face rendering of the target object can be used as driving human body signals.

[0070] Therefore, when the target object and the driving object are different, the face shape coefficient of the target object can be used to replace the face shape coefficient of the driving object in the driving video. This enables the construction of a driving human body signal based on the face shape coefficient of the target object and the driving video, so that the face shape in the generated video is the face shape of the target object. This can meet the video generation requirements when the target object and the driving object are different.

[0071] It should be noted that the aforementioned limbs may refer to the upper limbs or the four limbs, which can be determined based on the limb situation displayed in the driving video. This application does not limit this.

[0072] For example, a first facial expression coefficient can be obtained from a video or audio source other than the driving video. This first facial expression coefficient is then used to replace the second facial expression coefficient of the driving object in the first 3D facial model to obtain a third 3D facial model. This third 3D facial model is then rendered to obtain a third 3D facial rendering image. Finally, based on the key points of the torso and limbs in the driving object's human body key points, the 3D hand rendering image, and the third 3D facial rendering image, the driving human body signal is obtained. Here, the target object and the driving object can be the same or different; there is no limitation on this.

[0073] Among them, facial expression coefficients refer to the feature coefficients used to describe facial expressions in a 3D face model. Facial expression coefficients can be used to describe and simulate the movement of facial muscles and changes in facial expressions.

[0074] For example, if video A is the driving video, a 3D face model can be performed based on the image of the object contained in video B to obtain the 3D face model of the object in video B, thereby obtaining the facial expression coefficients of the object in video B. The facial expression coefficients of the object in video B can be used to replace the facial expression coefficients in the 3D face model of the driving object to obtain a new 3D face model. Then, a new 3D face rendering image is obtained through rendering. The key points of the torso and limbs of the driving object, the 3D hand rendering image of the driving object, and the new 3D face rendering image are used as driving human body signals.

[0075] Therefore, facial expression coefficients from other videos or audios can be used to replace the facial expression coefficients of the driving object, so that the body movements, head postures, etc. of the target object in the generated video are consistent with those of the driving object. The facial expressions of the target object are driven by other videos or audios, which can meet different video generation needs.

[0076] For example, reference human body signals can be obtained based on key points of the target object's torso and limbs, a 3D face rendering of the target object, and a 3D hand rendering of the target object in a reference image. For instance, the key points of the target object's torso and limbs, the 3D face rendering, and the 3D hand rendering can be used as reference human body signals.

[0077] Step 202: Dynamic feature extraction is performed on the reference human body signal and the driving human body signal respectively to obtain the first dynamic feature and the second dynamic feature.

[0078] In this application, step 202 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0079] Step 203: Extract texture features from the reference image to obtain the first texture feature.

[0080] In this application, step 203 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0081] Step 204: Calculate the first similarity matrix between the first dynamic feature and the second dynamic feature.

[0082] For example, the second dynamic feature can be used as the Query, the first dynamic feature as the Key, and the second dynamic feature and the first dynamic feature can be multiplied by a dot product to obtain the first similarity matrix.

[0083] Step 205: Scale and normalize the first similarity matrix to obtain the first attention weight matrix.

[0084] To avoid gradient vanishing or exploding due to excessively large dot product results during computation, the first similarity matrix can be scaled. This can be achieved by dividing by a scaling factor, such as the square root of the feature dimension.

[0085] After scaling the first similarity matrix, the softmax function can be used to normalize the scaled first similarity matrix to ensure that the sum of the elements in each row is 1, thus obtaining the first attention weight matrix.

[0086] Step 206: Based on the first attention weight matrix, the first texture features are weighted and summed to obtain the second texture features.

[0087] For example, the first texture feature can be considered to include multiple vectors, each of which can be used as a value vector. The first attention weight matrix can be used to perform a weighted summation on each value vector in the first texture feature, thereby applying the attention weights to the value vectors to obtain the second texture feature.

[0088] Step 207: The second texture feature, the second dynamic feature, and the noise are spliced ​​together to obtain the first spliced ​​feature.

[0089] In this application, step 207 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0090] Step 208: Perform denoising processing based on the first splicing feature to generate a target video displaying the target object.

[0091] In this application, step 208 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0092] In this embodiment, a first similarity matrix is ​​calculated between the first dynamic feature and the second dynamic feature, and the first similarity matrix is ​​scaled and normalized to obtain a first attention weight matrix. The first texture feature is weighted and summed using the first attention weight matrix, thereby dynamically adjusting the attention to the first texture feature based on the similarity between the second dynamic feature and the first dynamic feature. This achieves alignment between the texture of the reference image and the motion structure of the driving object, improving the accuracy of human motion in the generated video.

[0093] Figure 3 This is a flowchart illustrating a video generation method provided in another embodiment of this application.

[0094] like Figure 3 As shown, the video generation method includes:

[0095] Step 301: Obtain the reference human body signal corresponding to the reference image displaying the target object and the driving human body signal corresponding to the driving video.

[0096] In this application, step 301 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0097] Step 302: Dynamic feature extraction is performed on the reference human body signal and the driving human body signal respectively to obtain the first dynamic feature and the second dynamic feature.

[0098] In this application, step 302 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0099] Step 303: Extract texture features from the reference image to obtain the first texture feature.

[0100] In this application, step 303 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0101] Step 304: Perform cross-attention calculation on the first dynamic feature, the second dynamic feature, and the first texture feature to obtain the second texture feature.

[0102] In this application, step 304 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0103] Step 305: The second texture feature, the second dynamic feature, and the noise are spliced ​​together to obtain the first spliced ​​feature.

[0104] In this application, step 305 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0105] Step 306: The first dynamic feature and the first texture feature are concatenated to obtain the second concatenated feature.

[0106] For example, the first dynamic feature and the first texture feature corresponding to the reference image can be concatenated to obtain the second concatenated feature corresponding to the reference image, so that the concatenated feature contains the texture information and motion structure information of the target object.

[0107] Step 307: Extract texture features from the second splicing feature to obtain the third texture feature.

[0108] For example, the second splicing feature can be input into the appearance texture network, which can then further extract texture features.

[0109] For example, the appearance texture network can have the same structure as the denoising network, such as both the appearance texture network and the denoising network using the U-Net network.

[0110] For example, an appearance texture network may include appearance attention layers, each of which can perform self-attention computation on the input features, thereby obtaining third texture features at different scales through multiple appearance attention layers.

[0111] Step 308: Dynamic feature extraction is performed on the first splicing feature to obtain the third dynamic feature.

[0112] For example, the second dynamic feature, the second texture feature, and the first spliced ​​feature obtained by splicing noise can be input into a denoising network, which will process the data to obtain the third dynamic feature.

[0113] For example, the denoising network may include an appearance attention layer, which can perform self-attention calculation on the first concatenated features through the first appearance attention layer, and then perform self-attention calculation on the features input from the previous layer through subsequent appearance attention layers, thereby obtaining the third dynamic feature output by each appearance attention layer.

[0114] Step 309: Perform feature fusion based on the third texture feature and the third dynamic feature to obtain the first fused feature.

[0115] For example, the third texture feature can be fused with the third dynamic feature output by the corresponding layer in the denoising network to obtain the first fused feature.

[0116] For example, a second similarity matrix can be calculated between the third texture feature and the third dynamic feature. The second similarity matrix can then be scaled and normalized to obtain the corresponding attention weight matrix. Based on this attention weight matrix, the third texture feature can be weighted and summed to obtain the first fused feature.

[0117] To maximize the improvement in texture and structure alignment, when fusing the third texture features, the first similarity matrix of the first dynamic features and the second dynamic features can be used as a bias guide. Since the denoising network performs downsampling, the scale of the third dynamic features output by each layer is different from that of the first similarity matrix. Therefore, the first similarity matrix can be spatially scaled according to the size of the second similarity matrix to obtain the third similarity matrix, so that the scale of the third similarity matrix is ​​consistent with that of the second similarity matrix. Then, the second similarity matrix and the third similarity matrix are added to obtain the fourth similarity matrix. The fourth similarity matrix is ​​then scaled and normalized to obtain the second attention weight matrix. Finally, the third texture features are weighted and summed according to the second attention weight matrix to obtain the first fused feature.

[0118] Therefore, the first similarity matrix between the first dynamic feature and the second dynamic feature can be used as a bias guide. In the cross-attention calculation of each appearance attention layer in the denoising network, the similarity matrix between the third texture feature and the third dynamic feature can be added to the similarity matrix obtained by spatial scaling of the first similarity matrix. This can strengthen the position that needs to be aligned, give more attention, and improve the alignment effect between texture and motion structure.

[0119] Step 310: Generate the target video based on the first fusion feature.

[0120] For example, during the training phase, a hand texture feature library can be learned. The first fusion feature corresponding to each frame of the driving video and the learned hand texture feature library can be input into the hand attention layer for cross-attention calculation to obtain the second fusion feature corresponding to each frame. Then, the second fusion feature corresponding to each frame is input into the temporal attention layer for temporal attention calculation to obtain the third fusion feature. Finally, the target video is generated based on the third fusion feature.

[0121] For example, the size of the hand texture feature library can be N*C, where N can represent the number of texture samples and C can represent the feature dimension of each texture sample. It can be understood that the hand texture feature library includes N C-dimensional hand texture features.

[0122] For example, temporal attention computation can be performed based on the time dimension and channel dimension in the second fusion feature to capture the dependencies within the second fusion feature.

[0123] For example, the first fused feature can be used as the Query, and the hand texture feature library can be used as the Key and Value. The second fused feature can be obtained through attention calculation.

[0124] For example, the third fusion feature can be input into the next self-attention layer for self-attention, and then pass through the hand attention layer and temporal attention layer for attention calculation, until it passes through the last layer of the denoising network. Finally, the VAE decoder is used to decode the features output by the denoising network to generate the target video.

[0125] Therefore, cross-attention calculation is performed between the learned hand texture feature library and the first fusion feature, so that the obtained second fusion feature can include hand texture features, enriching the information of feature representation. Furthermore, temporal attention calculation is performed on the third fusion feature, which can capture the dependencies within the third fusion feature. Thus, a video is generated based on the obtained third fusion feature, which can ensure the temporal consistency between the generated video and the driving video, and can reduce the blur and jitter of the generated video.

[0126] To enhance and restore hand textures, a hand mask matrix can be used to specifically enhance and restore the hand texture. For example, a hand mask matrix can be obtained, and cross-attention calculation can be performed on the first fused feature and the hand texture feature library to obtain a first intermediate feature. Masking processing can be performed based on the hand mask matrix and the first intermediate feature to obtain a second intermediate feature. Finally, the second intermediate feature is added to the first fused feature to obtain a second fused feature.

[0127] The size of the hand mask matrix is ​​the same as that of the first intermediate feature. For example, in the hand mask matrix, the element value corresponding to the hand region can be 1, while the element value corresponding to other body parts besides the hand can be 0. When the hand mask matrix is ​​multiplied by the first intermediate feature, the feature values ​​of all other regions will be set to 0, except for the feature values ​​of the hand region, which remain unchanged. Thus, the second intermediate feature only contains the hand region feature, removing interference from other non-hand regions.

[0128] Therefore, the hand texture can be enhanced and restored by using a hand mask matrix. This can effectively repair the hand texture in the face of complex gesture generation problems in human body driving, thus greatly improving the overall quality of the generated video.

[0129] For example, when performing cross-attention calculation on the first fused feature and the hand texture feature library, a fifth similarity matrix between the first fused feature and the hand texture feature library can be calculated, and the fifth similarity matrix can be scaled and normalized to obtain a third attention weight matrix. Then, based on the third weight attention matrix, the hand texture feature library can be weighted and summed to obtain the first intermediate feature.

[0130] Therefore, by using the cross-attention mechanism, the deep interaction between the hand texture feature library and the first fused feature can be captured, and attention can be paid to hand texture information to improve the quality of the generated video.

[0131] To further improve the quality of the generated video, for example, the global features of the reference image can be used as global information. For instance, the CLIP (Contrastive Language-Image Pre-Training) model can be used to introduce global information. The CLIP model is a pre-trained model with two modalities: a text modality and a visual modality, corresponding to the text encoder and the image encoder, respectively.

[0132] For example, the image encoder in the CLIP model can be used to extract global features from the reference image to obtain global features. Self-attention calculation is then performed on the second stitched features to obtain a third intermediate feature. Cross-attention calculation is then performed between the third intermediate feature and the global features to obtain a third texture feature. Finally, feature fusion is performed based on the third texture feature and the third dynamic feature to obtain a first fused feature. The target video is then generated based on the first fused feature. Therefore, by introducing global features during texture feature extraction from the second stitched features, the extracted third texture feature incorporates global features, thereby improving the quality of the generated video.

[0133] For example, the cross-attention calculation between the third intermediate feature and the global feature can be performed in the following way: the sixth similarity matrix between the third intermediate feature and the global feature can be calculated, the sixth similarity matrix can be scaled and normalized to obtain the fourth attention weight matrix, and then the global feature can be weighted and summed according to the fourth attention weight matrix to obtain the third texture feature.

[0134] Therefore, by introducing global features of the target object and using a cross-attention mechanism, the deep interaction between texture features and global features can be captured, thereby improving the quality of the generated video.

[0135] For example, the first stitched feature can be input into a self-attention layer for self-attention calculation to obtain a fourth intermediate feature. This fourth intermediate feature is then input into a multimodal attention layer for cross-attention calculation with global features of the reference image to obtain a third dynamic feature. Finally, feature fusion is performed based on the third texture feature and the third dynamic feature to obtain a first fused feature. The target video is then generated based on this first fused feature. Thus, by introducing global features during dynamic feature extraction from the first stitched feature, the extracted third dynamic feature incorporates global features, thereby improving the quality of the generated video.

[0136] For example, the cross-attention calculation between the fourth intermediate feature and the global feature can be performed in the following way: the seventh similarity matrix between the fourth intermediate feature and the global feature can be calculated, the seventh similarity matrix can be scaled and normalized to obtain the fifth attention weight matrix, and then the global feature can be weighted and summed according to the fifth attention weight matrix to obtain the third dynamic feature.

[0137] Therefore, by introducing global features of the target object and using a cross-attention mechanism, the deep interaction between dynamic features and global features can be captured, thereby improving the quality of the generated video.

[0138] For example, the denoising network may include a self-attention layer, a multimodal attention layer, an appearance attention layer, a hand attention layer, and a temporal attention layer. The self-attention layer can perform self-attention calculation on the features input from the previous layer. The features output from the self-attention layer can be input into the next layer, the multimodal attention layer, and perform cross-attention calculation with the global features of the reference image. The features output from the multimodal attention layer are input into the next layer, the appearance attention layer, and perform cross-attention calculation with the third texture features. The features output from the appearance attention layer are input into the next layer, the hand attention layer, and perform cross-attention calculation with the hand texture feature library. The features output from the hand attention layer are input into the next layer, the temporal attention layer, and perform temporal attention calculation. The features output from the last layer of the denoising network can be input into the VAE decoder for decoding to generate the target video.

[0139] In this embodiment, texture features can be extracted by concatenating the first dynamic feature and the first texture feature of the reference image to obtain the second concatenated feature, resulting in the third texture feature. Dynamic features are then extracted from the first concatenated feature corresponding to the driving object to obtain the third dynamic feature. The third texture feature and the third dynamic feature are then fused. Based on the fused feature, a video is generated. Thus, a video is generated based on the texture feature corresponding to the reference image and the dynamic feature corresponding to the driving object. This allows the generated video to contain both the texture information of the reference image and the action of the target object to be consistent with the action of the driving object, thereby improving the quality of the generated video.

[0140] To implement the above embodiments, the following is combined with Figure 4 To explain, Figure 4 This is a schematic diagram of a video generation process provided in an embodiment of this application.

[0141] For human-driven videos with more complex motion patterns, a precise human-driven signal can be proposed based on a combination of 2D and 3D methods. Specifically, for coarse-grained motion parts such as the upper limbs and torso, sparser 2D keypoints can be used for representation; while for fine-grained motion parts such as the face and hands, more accurate 3D face and hand reconstruction renderings can be used. Compared to 2D keypoints, 3D renderings can better represent motion postures and occlusion relationships, and have a more accurate ability to express complex gestures and frequently moving lip movements.

[0142] The human-driven learning process can be divided into two branches: motion structure and appearance texture. For example... Figure 4 As shown, this application employs a dual-path U-Net network structure based on a diffusion model. The upper branch, the appearance and texture network, learns appearance and texture information based on randomly selected images, while the lower branch, the denoising network, learns motion structure information based on the input driving human body signals. Both branches use the same U-Net convolutional structure, and a cross-attention mechanism is used for information exchange on the output features of corresponding layers.

[0143] In this embodiment, a pre-trained large-scale VAE model can be used as a local feature extractor and decoder in texture learning; simultaneously, a dynamic feature encoder is used to characterize the motion structure information driving the human body signal. Furthermore, a CLIP model can be used to introduce global information into the diffusion model, such as... Figure 4 As shown, the CLIP encoder can be used to extract global features from the reference image to obtain global features f. r c .

[0144] Since the human pose driving the human body signal is usually different from the human pose in the reference image, the misalignment between texture and structural information leads to inaccurate texture in the generated image. This application proposes a dynamic information enhancement texture alignment technique based on an attention mechanism. Figure 4 As shown, a second dynamic feature F is extracted using a dynamic feature encoder for the input driving human body signal and the reference human body signal corresponding to the reference image. i and the first dynamic feature F r The two dynamic feature sizes can be H*W*C. Simultaneously, the first texture feature Z of the reference image is extracted using a VAE encoder. r By separately F i Fr and Z r Using the attention mechanism as the query, key, and value, the structure-aligned second texture feature Z can be obtained. i p Attention calculation Figure 4 The specific calculation of the dynamic information-enhanced texture alignment shown in the figure, followed by the second texture feature Z i p With the second dynamic feature F i Gaussian noise is concatenated and used as the input to the U-Net denoising network.

[0145] Since the U-Net performs cross-attention mechanism calculations in different layers for both appearance texture and motion structure branches, this embodiment uses a second dynamic feature F to maximize the improvement in texture-structure alignment. i and the first dynamic feature F r Similarity matrix B i As a bias-guided approach, a reference image bias attention mechanism is proposed. Specifically, in the cross-attention computation of each layer of the two U-Nets, the similarity matrices obtained from texture features and dynamic features are compared with the reference image B. i By summing the similarity matrices obtained through spatial scaling, the positions that need to be aligned can be strengthened, thus giving them more attention.

[0146] To address the problem of hand texture synthesis, a hand restoration module was designed, comprising a learnable hand texture feature library with a feature size of N*C, and a hand attention mechanism for optimizing hand texture. In the hand attention mechanism, the output feature map of any layer of the denoising network branch U-Net is used as the query, and the hand texture feature library is used as the key and value. Additionally, a hand mask matrix can be used to ensure that the hand attention calculation result is added only to the original output feature map in the hand region, specifically enhancing and restoring the hand texture. The use of the hand mask matrix also restricts the information learned by the hand texture feature library to be only related to the hand, preventing the leakage of other redundant information.

[0147] like Figure 4 As shown, the first texture feature Z can be... r With the first dynamic feature F r The concatenated data is then input into the appearance texture network, while simultaneously introducing global features f. r c Both the appearance texture network and the denoising network contain self-attention layers and multimodal attention layers. The self-attention layer performs self-attention calculation on the features input from the previous layer, and the features output from the self-attention layer are input into the multimodal attention layer and the global features f. r cCross-attention calculation is performed. Then, the features output from the corresponding multimodal attention layers in the appearance texture network and the denoising network are input into the appearance attention layer for cross-attention calculation. During the cross-attention calculation in the appearance attention layer, the similarity matrix of the features output from the two multimodal attention layers can be compared with the similarity matrix B. i The similarity matrices obtained by spatial scaling are added to enhance the positions that need to be aligned. The features output by the appearance attention layer and the learned hand texture feature library are input into the hand attention layer for cross-attention calculation. The features output by the hand attention layer are input into the temporal attention layer for temporal attention calculation. Finally, the features input from the denoising network are input into the VAE decoder for decoding to generate the video.

[0148] During the training phase of the diffusion model, after learning the appearance texture and motion structure at the image level, the parameters of the appearance texture and motion structure branch network structures can be fixed. An additional temporal attention layer can be added separately to the denoising network branch to train the temporal modeling capability of the diffusion model. Training the temporal attention layer can significantly reduce blur and jitter in video footage.

[0149] The video generation method in this application can construct a driving human signal suitable for precise human motion based on the dual-path diffusion model generation framework, and proposes a reference image bias attention mechanism. Based on this driving human signal, dynamic texture alignment is performed to achieve effective alignment between the driving human signal and the reference image. Simultaneously, based on a learnable hand texture feature library method and a cross-attention mechanism, we effectively intrude textures to address the complex gesture generation problem in human motion, significantly improving the overall generation quality.

[0150] To implement the above embodiments, this application also proposes a video generation apparatus. Figure 5 This is a schematic diagram of the structure of a video generation apparatus provided in an embodiment of this application.

[0151] like Figure 5 As shown, the video generation apparatus 500 includes:

[0152] The first acquisition module 510 is used to acquire a reference human body signal corresponding to a reference image displaying a target object and a driving human body signal corresponding to a driving video; wherein, the reference human body signal is used to indicate the human body posture of the target object, and the driving human body signal is used to indicate the human body posture of the driving object in the driving video.

[0153] The dynamic feature extraction module 520 is used to extract dynamic features from the reference human body signal and the driving human body signal respectively to obtain the first dynamic feature and the second dynamic feature.

[0154] The texture feature extraction module 530 is used to extract texture features from the reference image to obtain a first texture feature;

[0155] The second acquisition module 540 is used to perform cross-attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain the second texture feature;

[0156] The feature splicing module 550 is used to splice the second texture feature, the second dynamic feature and noise to obtain the first spliced ​​feature;

[0157] The generation module 560 is used to perform noise reduction processing based on the first splicing features to generate a target video displaying the target object.

[0158] Optionally, the second acquisition module 540 is used for:

[0159] Calculate the first similarity matrix between the first dynamic feature and the second dynamic feature;

[0160] The first similarity matrix is ​​scaled and normalized to obtain the first attention weight matrix;

[0161] The first texture features are weighted and summed according to the first attention weight matrix to obtain the second texture features.

[0162] Optionally, the generation module 560 is configured to:

[0163] The first dynamic feature and the first texture feature are concatenated to obtain the second concatenated feature;

[0164] Texture features are extracted from the second splicing feature to obtain the third texture feature;

[0165] Dynamic feature extraction is performed on the first splicing feature to obtain the third dynamic feature;

[0166] Based on the third texture feature and the third dynamic feature, feature fusion is performed to obtain the first fused feature;

[0167] The target video is generated based on the first fusion feature.

[0168] Optionally, the generation module 560 is configured to:

[0169] Obtain the first similarity matrix between the first dynamic feature and the second dynamic feature;

[0170] Calculate the second similarity matrix between the third texture feature and the third dynamic feature;

[0171] The first similarity matrix is ​​processed according to the size of the second similarity matrix to obtain a third similarity matrix; wherein the size of the third similarity matrix is ​​the same as the size of the second similarity matrix;

[0172] The third similarity matrix is ​​added to the second similarity matrix to obtain the fourth similarity matrix;

[0173] The fourth similarity matrix is ​​scaled and normalized to obtain the second attention weight matrix;

[0174] The third texture feature is weighted and summed according to the second attention weight matrix to obtain the first fused feature.

[0175] Optionally, the generation module 560 is configured to:

[0176] The second fusion feature is obtained by performing cross-attention calculation based on the first fusion feature and the learned hand texture feature library;

[0177] Temporal attention calculation is performed on the second fusion feature to obtain the third fusion feature;

[0178] The target video is generated based on the third fusion feature.

[0179] Optionally, the generation module 560 is configured to:

[0180] Obtain the hand mask matrix;

[0181] Cross-attention calculation is performed on the first fused feature and the hand texture feature library to obtain the first intermediate feature;

[0182] Based on the hand mask matrix, the first intermediate feature is masked to obtain the second intermediate feature;

[0183] The second intermediate feature is added to the first fusion feature to obtain the second fusion feature.

[0184] Optionally, the generation module 560 is configured to:

[0185] Calculate the fifth similarity matrix between the first fused feature and the hand texture feature library;

[0186] The fifth similarity matrix is ​​scaled and normalized to obtain the third attention weight matrix;

[0187] The first intermediate feature is obtained by weighting and summing the hand texture feature library according to the third attention weight matrix.

[0188] Optionally, the generation module 560 is configured to:

[0189] Global feature extraction is performed on the reference image to obtain global features;

[0190] Self-attention calculation is performed on the second spliced ​​feature to obtain the third intermediate feature;

[0191] The third texture feature is obtained by performing cross-attention calculation on the third intermediate feature and the global feature.

[0192] Optionally, the generation module 560 is configured to:

[0193] Calculate the sixth similarity matrix between the third intermediate feature and the global feature;

[0194] The sixth similarity matrix is ​​scaled and normalized to obtain the fourth attention weight matrix;

[0195] The third texture feature is obtained by weighting and summing the global features according to the fourth attention weight matrix.

[0196] Optionally, the generation module 560 is configured to:

[0197] Global feature extraction is performed on the reference image to obtain global features;

[0198] Self-attention calculation is performed on the first spliced ​​feature to obtain the fourth intermediate feature;

[0199] The third dynamic feature is obtained by performing cross-attention calculation on the fourth intermediate feature and the global feature.

[0200] Optionally, the generation module 560 is configured to:

[0201] Calculate the seventh similarity matrix between the fourth intermediate feature and the global feature;

[0202] The seventh similarity matrix is ​​scaled and normalized to obtain the fifth attention weight matrix;

[0203] The third dynamic feature is obtained by weighting and summing the global features according to the fifth attention weight matrix.

[0204] Optionally, the first acquisition module 510 is configured to:

[0205] Key point detection is performed on each frame of the driving video to obtain the human key points of the driving object;

[0206] Based on each frame of the driving video, the first three-dimensional face model and the three-dimensional hand model of the driving object are obtained.

[0207] The first 3D face model and the 3D hand model are rendered to obtain a first 3D face rendering image and a 3D hand rendering image;

[0208] The driving human body signal is obtained based on the key points of the torso and limbs in the human body key points, the first three-dimensional face rendering image and the three-dimensional hand rendering image.

[0209] Optionally, the first acquisition module is configured to:

[0210] Key point detection is performed on each frame of the driving video to obtain the human key points of the driving object;

[0211] Based on each frame of the driving video, the first 3D face model and 3D hand model of the driving object are obtained.

[0212] Obtain the first face shape coefficient of the target object, and replace the second face shape coefficient of the driving object in the first three-dimensional face model with the first face shape coefficient to obtain the second three-dimensional face model;

[0213] The second 3D face model and the 3D hand model are rendered to obtain a second 3D face rendering image and a 3D hand rendering image;

[0214] The driving human body signal is obtained based on the key points of the torso and limbs in the human body key points, the second three-dimensional face rendering image and the three-dimensional hand rendering image.

[0215] Optionally, the first acquisition module 510 is configured to:

[0216] Key point detection is performed on each frame of the driving video to obtain the human key points of the driving object;

[0217] Based on each frame of the driving video, the first 3D face model and 3D hand model of the driving object are obtained.

[0218] A first facial expression coefficient is obtained, and the second facial expression coefficient of the driving object in the first three-dimensional facial model is replaced with the first facial expression coefficient to obtain a third three-dimensional facial model; wherein, the first facial expression coefficient is obtained from other videos or audios besides the driving video;

[0219] The third 3D face model and the 3D hand model are rendered to obtain a third 3D face rendering image and a 3D hand rendering image;

[0220] The driving human body signal is obtained based on the key points of the torso and limbs in the human body key points, the third three-dimensional face rendering image, and the three-dimensional hand rendering image.

[0221] It should be noted that the explanation of the aforementioned video generation method embodiment also applies to the video generation device of this embodiment, and therefore will not be repeated here.

[0222] In this embodiment, by performing dynamic feature encoding on the reference human body signal corresponding to the reference image and the driving human body signal corresponding to the driving video, a first dynamic feature and a second dynamic feature are obtained. Texture features are extracted from the reference image to obtain a first texture feature. Cross-attention calculation is performed on the first dynamic feature, the second dynamic feature, and the first texture feature. Through the attention mechanism, a second texture feature that aligns the texture with the motion structure is obtained, thus achieving effective alignment between the driving human body signal of the driving object and the reference image. Based on the second texture feature and the second dynamic feature, the target video is generated, improving the accuracy of the texture in the generated video and improving the quality of the generated video.

[0223] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0224] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0225] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 602 or a computer program loaded from storage unit 608 into RAM (Random Access Memory) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. I / O (Input / Output) interface 605 is also connected to bus 604.

[0226] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0227] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as video generation methods. For example, in some embodiments, the video generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the video generation method by any other suitable means (e.g., by means of firmware).

[0228] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0229] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0230] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0231] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0232] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0233] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0234] According to an embodiment of this application, this application also provides a computer program product that, when an instruction processor in the computer program product is executed, performs the video generation method proposed in the above embodiments of this application.

[0235] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0236] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video generation method, comprising: Acquire a reference human body signal corresponding to a reference image displaying a target object and a driving human body signal corresponding to a driving video; wherein, the reference human body signal is used to indicate the human posture of the target object, and the driving human body signal is used to indicate the human posture of the driven object in the driving video; Dynamic features are extracted from the reference human body signal and the driving human body signal respectively to obtain the first dynamic feature and the second dynamic feature; The reference image is subjected to texture feature extraction to obtain the first texture feature; Cross-attention calculation is performed on the first dynamic feature, the second dynamic feature, and the first texture feature to obtain the second texture feature; The second texture feature, the second dynamic feature, and the noise are spliced ​​together to obtain the first spliced ​​feature; The first dynamic feature and the first texture feature are concatenated to obtain the second concatenated feature; Texture features are extracted from the second splicing feature to obtain the third texture feature; Dynamic feature extraction is performed on the first splicing feature to obtain the third dynamic feature; Based on the third texture feature and the third dynamic feature, feature fusion is performed to obtain the first fused feature; Based on the first fusion feature, a target video displaying the target object is generated.

2. The method as described in claim 1, wherein, The step of performing cross-attention calculation on the first dynamic feature, the second dynamic feature, and the first texture feature to obtain the second texture feature includes: Calculate the first similarity matrix between the first dynamic feature and the second dynamic feature; The first similarity matrix is ​​scaled and normalized to obtain the first attention weight matrix; The first texture features are weighted and summed according to the first attention weight matrix to obtain the second texture features.

3. The method as described in claim 1, wherein, The step of fusing features based on the third texture feature and the third dynamic feature to obtain the first fused feature includes: Obtain the first similarity matrix between the first dynamic feature and the second dynamic feature; Calculate the second similarity matrix between the third texture feature and the third dynamic feature; The first similarity matrix is ​​spatially scaled according to the size of the second similarity matrix to obtain a third similarity matrix; wherein the size of the third similarity matrix is ​​the same as the size of the second similarity matrix; The third similarity matrix is ​​added to the second similarity matrix to obtain the fourth similarity matrix; The fourth similarity matrix is ​​scaled and normalized to obtain the second attention weight matrix; The third texture feature is weighted and summed according to the second attention weight matrix to obtain the first fused feature.

4. The method of claim 1, wherein, The step of generating a target video displaying the target object based on the first fusion feature includes: The second fusion feature is obtained by performing cross-attention calculation based on the first fusion feature and the learned hand texture feature library; Temporal attention calculation is performed on the second fusion feature to obtain the third fusion feature; The target video is generated based on the third fusion feature.

5. The method of claim 4, wherein, The step of performing cross-attention calculation based on the first fused feature and the learned hand texture feature library to obtain the second fused feature includes: Obtain the hand mask matrix; Cross-attention calculation is performed on the first fused feature and the hand texture feature library to obtain the first intermediate feature; Based on the hand mask matrix, the first intermediate feature is masked to obtain the second intermediate feature; The second intermediate feature is added to the first fusion feature to obtain the second fusion feature.

6. The method of claim 5, wherein, The step of performing cross-attention calculation on the first fused feature and the hand texture feature library to obtain the first intermediate feature includes: Calculate the fifth similarity matrix between the first fused feature and the hand texture feature library; The fifth similarity matrix is ​​scaled and normalized to obtain the third attention weight matrix; The first intermediate feature is obtained by weighting and summing the hand texture feature library according to the third attention weight matrix.

7. The method of claim 1, wherein, The step of extracting texture features from the second splicing feature to obtain the third texture feature includes: Global feature extraction is performed on the reference image to obtain global features; Self-attention calculation is performed on the second spliced ​​feature to obtain the third intermediate feature; The third texture feature is obtained by performing cross-attention calculation on the third intermediate feature and the global feature.

8. The method of claim 7, wherein, The step of performing cross-attention calculation on the third intermediate feature and the global feature to obtain the third texture feature includes: Calculate the sixth similarity matrix between the third intermediate feature and the global feature; The sixth similarity matrix is ​​scaled and normalized to obtain the fourth attention weight matrix; The third texture feature is obtained by weighting and summing the global features according to the fourth attention weight matrix.

9. The method of claim 1, wherein, The step of dynamically extracting the first splicing feature to obtain the third dynamic feature includes: Global feature extraction is performed on the reference image to obtain global features; Self-attention calculation is performed on the first spliced ​​feature to obtain the fourth intermediate feature; The third dynamic feature is obtained by performing cross-attention calculation on the fourth intermediate feature and the global feature.

10. The method of claim 9, wherein, The step of performing cross-attention calculation on the fourth intermediate feature and the global feature to obtain the third dynamic feature includes: Calculate the seventh similarity matrix between the fourth intermediate feature and the global feature; The seventh similarity matrix is ​​scaled and normalized to obtain the fifth attention weight matrix; The third dynamic feature is obtained by weighting and summing the global features according to the fifth attention weight matrix.

11. The method according to any one of claims 1-10, wherein, The acquisition of the driving human body signal corresponding to the driving video includes: Key point detection is performed on each frame of the driving video to obtain the human key points of the driving object; Based on each frame of the driving video, the first three-dimensional face model and the three-dimensional hand model of the driving object are obtained; The first 3D face model and the 3D hand model are rendered to obtain a first 3D face rendering image and a 3D hand rendering image; The driving human body signal is obtained based on the key points of the torso and limbs in the human body key points, the first three-dimensional face rendering image and the three-dimensional hand rendering image.

12. The method according to any one of claims 1-10, wherein, The acquisition of the driving human body signal corresponding to the driving video includes: Key point detection is performed on each frame of the driving video to obtain the human key points of the driving object; Based on each frame of the driving video, the first three-dimensional face model and the three-dimensional hand model of the driving object are obtained; Obtain the first face shape coefficient of the target object, and replace the second face shape coefficient of the driving object in the first three-dimensional face model with the first face shape coefficient to obtain the second three-dimensional face model; The second 3D face model and the 3D hand model are rendered to obtain a second 3D face rendering image and a 3D hand rendering image; The driving human body signal is obtained based on the key points of the torso and limbs in the human body key points, the second three-dimensional face rendering image and the three-dimensional hand rendering image.

13. The method according to any one of claims 1-10, wherein, The acquisition of the driving human body signal corresponding to the driving video includes: Key point detection is performed on each frame of the driving video to obtain the human key points of the driving object; Based on each frame of the driving video, the first three-dimensional face model and the three-dimensional hand model of the driving object are obtained; A first facial expression coefficient is obtained, and the second facial expression coefficient of the driving object in the first three-dimensional facial model is replaced with the first facial expression coefficient to obtain a third three-dimensional facial model; wherein, the first facial expression coefficient is obtained from other videos or audios besides the driving video; The third 3D face model and the 3D hand model are rendered to obtain a third 3D face rendering image and a 3D hand rendering image; The driving human body signal is obtained based on the key points of the torso and limbs in the human body key points, the third three-dimensional face rendering image, and the three-dimensional hand rendering image.

14. A video generation apparatus, comprising: The first acquisition module is used to acquire a reference human body signal corresponding to a reference image displaying a target object and a driving human body signal corresponding to a driving video; wherein, the reference human body signal is used to indicate the human body posture of the target object, and the driving human body signal is used to indicate the human body posture of the driving object in the driving video; The dynamic feature extraction module is used to extract dynamic features from the reference human body signal and the driving human body signal respectively to obtain the first dynamic feature and the second dynamic feature. The texture feature extraction module is used to extract texture features from the reference image to obtain the first texture feature; The second acquisition module is used to perform cross-attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain the second texture feature; The feature splicing module is used to splice the second texture feature, the second dynamic feature, and noise to obtain the first spliced ​​feature; The generation module is used to perform noise reduction processing based on the first splicing features to generate a target video displaying the target object. The generation module is used for: The first dynamic feature and the first texture feature are concatenated to obtain the second concatenated feature; Texture features are extracted from the second splicing feature to obtain the third texture feature; Dynamic feature extraction is performed on the first splicing feature to obtain the third dynamic feature; Based on the third texture feature and the third dynamic feature, feature fusion is performed to obtain the first fused feature; The target video is generated based on the first fusion feature.

15. The apparatus of claim 14, wherein, The second acquisition module is used for: Calculate the first similarity matrix between the first dynamic feature and the second dynamic feature; The first similarity matrix is ​​scaled and normalized to obtain the first attention weight matrix; The first texture features are weighted and summed according to the first attention weight matrix to obtain the second texture features.

16. The apparatus of claim 14, wherein, The generation module is used for: Obtain the first similarity matrix between the first dynamic feature and the second dynamic feature; Calculate the second similarity matrix between the third texture feature and the third dynamic feature; The first similarity matrix is ​​processed according to the size of the second similarity matrix to obtain a third similarity matrix; wherein the size of the third similarity matrix is ​​the same as the size of the second similarity matrix; Add the first similarity matrix to the third similarity matrix to obtain the fourth similarity matrix; The fourth similarity matrix is ​​scaled and normalized to obtain the second attention weight matrix; The third texture feature is weighted and summed according to the second attention weight matrix to obtain the first fused feature.

17. The apparatus of claim 14, wherein, The generation module is used for: The second fusion feature is obtained by performing cross-attention calculation based on the first fusion feature and the learned hand texture feature library; Temporal attention calculation is performed on the second fusion feature to obtain the third fusion feature; The target video is generated based on the third fusion feature.

18. The apparatus of claim 17, wherein, The generation module is used for: Obtain the hand mask matrix; Cross-attention calculation is performed on the first fused feature and the hand texture feature library to obtain the first intermediate feature; Based on the hand mask matrix, the first intermediate feature is masked to obtain the second intermediate feature; The second intermediate feature is added to the first fusion feature to obtain the second fusion feature.

19. The apparatus of claim 18, wherein, The generation module is used for: Calculate the fifth similarity matrix between the first fused feature and the hand texture feature library; The fifth similarity matrix is ​​scaled and normalized to obtain the third attention weight matrix; The first intermediate feature is obtained by weighting and summing the hand texture feature library according to the third attention weight matrix.

20. The apparatus of claim 14, wherein, The generation module is used for: Global feature extraction is performed on the reference image to obtain global features; Self-attention calculation is performed on the second spliced ​​feature to obtain the third intermediate feature; The third texture feature is obtained by performing cross-attention calculation on the third intermediate feature and the global feature.

21. The apparatus of claim 20, wherein, The generation module is used for: Calculate the sixth similarity matrix between the third intermediate feature and the global feature; The sixth similarity matrix is ​​scaled and normalized to obtain the fourth attention weight matrix; The third texture feature is obtained by weighting and summing the global features according to the fourth attention weight matrix.

22. The apparatus of claim 14, wherein, The generation module is used for: Global feature extraction is performed on the reference image to obtain global features; Self-attention calculation is performed on the first spliced ​​feature to obtain the fourth intermediate feature; The third dynamic feature is obtained by performing cross-attention calculation on the fourth intermediate feature and the global feature.

23. The apparatus of claim 22, wherein, The generation module is used for: Calculate the seventh similarity matrix between the fourth intermediate feature and the global feature; The seventh similarity matrix is ​​scaled and normalized to obtain the fifth attention weight matrix; The third dynamic feature is obtained by weighting and summing the global features according to the fifth attention weight matrix.

24. The apparatus according to any one of claims 14-23, wherein, The first acquisition module is used for: Key point detection is performed on each frame of the driving video to obtain the human key points of the driving object; Based on each frame of the driving video, the first three-dimensional face model and the three-dimensional hand model of the driving object are obtained; The first 3D face model and the 3D hand model are rendered to obtain a first 3D face rendering image and a 3D hand rendering image; The driving human body signal is obtained based on the key points of the torso and limbs in the human body key points, the first three-dimensional face rendering image and the three-dimensional hand rendering image.

25. The apparatus according to any one of claims 14-23, wherein, The first acquisition module is used for: Key point detection is performed on each frame of the driving video to obtain the human key points of the driving object; Based on each frame of the driving video, the first three-dimensional face model and the three-dimensional hand model of the driving object are obtained; Obtain the first face shape coefficient of the target object, and replace the second face shape coefficient of the driving object in the first three-dimensional face model with the first face shape coefficient to obtain the second three-dimensional face model; The second 3D face model and the 3D hand model are rendered to obtain a second 3D face rendering image and a 3D hand rendering image; The driving human body signal is obtained based on the key points of the torso and limbs in the human body key points, the second three-dimensional face rendering image and the three-dimensional hand rendering image.

26. The apparatus according to any one of claims 14-23, wherein, The first acquisition module is used for: Key point detection is performed on each frame of the driving video to obtain the human key points of the driving object; Based on each frame of the driving video, the first three-dimensional face model and the three-dimensional hand model of the driving object are obtained; A first facial expression coefficient is obtained, and the second facial expression coefficient of the driving object in the first three-dimensional facial model is replaced with the first facial expression coefficient to obtain a third three-dimensional facial model; wherein, the first facial expression coefficient is obtained from other videos or audios besides the driving video; The third 3D face model and the 3D hand model are rendered to obtain a third 3D face rendering image and a 3D hand rendering image; The driving human body signal is obtained based on the key points of the torso and limbs in the human body key points, the third three-dimensional face rendering image, and the three-dimensional hand rendering image.

27. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

29. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-13.

Images