Video generation method and device, electronic equipment and storage medium
By acquiring and processing human signals, combining cross attention mechanism and denoising network, videos with the human posture and movement of the target object are generated, which solves the problem of insufficient accuracy of video generation texture and movement in the prior art, and achieves high-quality Video generation.
Patent Information
- Application Number
- CN202411267110.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-10
AI Technical Summary
The prior art is difficult to effectively generate videos with human postures and actions of the target object, especially when the target object is different from the driving object, the texture and action accuracy of the generated videos are insufficient.
By acquiring the human body signals in the reference image and driving video, dynamic feature extraction and texture feature extraction are performed, and texture and motion structure are aligned with the cross attention mechanism to generate target videos, and the denoising network and VAE model are used to improve the quality of video generation.
It realizes video generation where the target object and the driver object actions are consistent, improves texture accuracy and quality in video generation, and is suitable for video generation scenarios of different objects.
Smart Images

Figure CN119031204A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC (Artificial Intelligence Generated Content), digital humans, and smart e-commerce, and specifically to a video generation method, device, electronic device, and storage medium. Background Art
[0002] Human body driving technology refers to the technology of driving the generation of a target object's limb movement video based on any video of human body movement, wherein the target person in the generated human body driving video has the same limb movement, gestures, facial expressions and lip movements as the driving person. Summary of the invention
[0003] The present application provides a video generation method, device, electronic device and storage medium.
[0004] According to one aspect of the present application, a video generation method is provided, comprising:
[0005] Acquire a reference human body signal corresponding to a reference image showing a target object and a driving human body signal corresponding to a driving video; wherein the reference human body signal is used to indicate a human body posture of the target object, and the driving human body signal is used to indicate a human body posture of the driving object in the driving video;
[0006] Extracting dynamic features from the reference human body signal and the driving human body signal respectively to obtain a first dynamic feature and a second dynamic feature;
[0007] Extracting texture features from the reference image to obtain first texture features;
[0008] Performing cross attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain a second texture feature;
[0009] splicing the second texture feature, the second dynamic feature and the noise to obtain a first splicing feature;
[0010] De-noising is performed according to the first splicing feature to generate a target video showing the target object.
[0011] According to another aspect of the present application, a video generating device is provided, comprising:
[0012] A first acquisition module is used to acquire a reference human body signal corresponding to a reference image showing a target object and a driving human body signal corresponding to a driving video; wherein the reference human body signal is used to indicate a human body posture of the target object, and the driving human body signal is used to indicate a human body posture of the driving object in the driving video;
[0013] A dynamic feature extraction module, used to extract dynamic features from the reference human body signal and the driving human body signal respectively to obtain a first dynamic feature and a second dynamic feature;
[0014] A texture feature extraction module, used to extract texture features from the reference image to obtain a first texture feature;
[0015] A second acquisition module, configured to perform cross-attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain a second texture feature;
[0016] A feature splicing module, used for splicing the second texture feature, the second dynamic feature and the noise to obtain a first splicing feature;
[0017] A generation module is used to perform denoising processing according to the first splicing feature to generate a target video showing the target object.
[0018] According to another aspect of the present application, there is provided an electronic device, including:
[0019] at least one processor; and
[0020] a memory communicatively connected to the at least one processor; wherein,
[0021] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method described in the above embodiment.
[0022] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to the above embodiment.
[0023] According to another aspect of the present application, a computer program product is provided, including a computer program, wherein the computer program implements the steps of the method described in the above embodiment when executed by a processor.
[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0026] Figure 1 A schematic diagram of a flow chart of a video generation method provided in an embodiment of the present application;
[0027] Figure 2 A schematic diagram of a flow chart of a video generation method provided by another embodiment of the present application;
[0028] Figure 3 A schematic diagram of a flow chart of a video generation method provided by another embodiment of the present application;
[0029] Figure 4 A schematic diagram of a video generation process provided in an embodiment of the present application;
[0030] Figure 5 A schematic diagram of the structure of a video generating device provided in one embodiment of the present application;
[0031] Figure 6 It is a block diagram of an electronic device used to implement the video generation method of an embodiment of the present application. DETAILED DESCRIPTION
[0032] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0033] The video generation method, device, electronic device and storage medium of the embodiments of the present application are described below with reference to the accompanying drawings.
[0034] Figure 1 A flowchart of a video generation method provided in one embodiment of the present application.
[0035] The video generating method of the embodiment of the present application may be executed by the video generating device of the embodiment of the present application, and the device may be configured in an electronic device.
[0036] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.
[0037] like Figure 1 As shown, the video generation method includes:
[0038] Step 101 , obtaining a reference human body signal corresponding to a reference image showing a target object and a driving human body signal corresponding to a driving video.
[0039] The reference human body signal can be used to indicate the human body posture of the target object in the reference image, and the driving human body signal can be used to indicate the human body posture of the driving object in the driving video. In addition, the target object and the driving object can be the same or different, and this application does not limit this.
[0040] Exemplarily, key point detection may be performed on a reference image to obtain human body key point information of the target object, and based on the human body key point information of the target object, a posture heat map of the target object may be obtained, and the posture heat map of the target object may be used as a reference human body signal.
[0041] Exemplarily, if the reference image is a frame image extracted from a driving video, then the target object is the same as the driving object, and key point detection can be performed on each frame image in the driving video. Based on the human body key point information of the driving object in each frame image, a posture heat map corresponding to the reference in each frame image is obtained, and the posture heat map of the driving object in each frame image is used as a driving human body signal.
[0042] Since fingers may be occluded during hand movements and lip movements are relatively frequent, for example, coarse-grained moving parts such as limbs and torso can be represented by relatively sparse 2D key points, and fine-grained moving parts such as face and hands can be represented by more accurate 3D face and 3D hand reconstructed renderings. In other words, 2D key points of the human torso and limbs, three-dimensional renderings of hands and faces, etc. can be used as driving human body signals.
[0043] Compared with 2D key points, 3D renderings can better represent motion postures and occlusion relationships, and have more accurate expression capabilities for complex gestures and frequent lip movements. Therefore, fine-grained areas such as hands and faces are represented by 3D renderings, which can improve the accuracy of driving human body signals, thereby generating videos based on driving human body signals, which can improve the accuracy of human body movements in the generated videos. In addition, coarse-grained areas such as limbs and torso are represented by 2D key points, which can reduce the amount of calculation and improve the efficiency of video generation while ensuring the accuracy of torso and limb movements.
[0044] Step 102 , extracting dynamic features from the reference human body signal and the driving human body signal respectively to obtain a first dynamic feature and a second dynamic feature.
[0045] Exemplarily, a dynamic feature encoder may be used to extract dynamic features from a reference human body signal to obtain a first dynamic feature, and a dynamic feature encoder may be used to extract dynamic features from a driving human body signal to obtain a second dynamic feature.
[0046] Exemplarily, the size of the first dynamic feature and the second dynamic feature may be H*W*C, where H, W, and C represent height, width, and the number of channels, respectively.
[0047] Step 103: extract texture features from the reference image to obtain first texture features.
[0048] VAE (Variational AutoEncoder) is a generative model that combines the ideas of autoencoder and variational Bayesian inference. VAE can learn the latent representation of input data and generate new and similar data samples based on the latent representation. VAE includes an encoder and a decoder. The encoder in VAE can map the input data to a point in the latent space, i.e., the latent representation, and the decoder in VAE can generate new and similar data samples based on the latent representation.
[0049] Exemplarily, the reference image may be input into a VAE encoder, and the VAE encoder may be used to extract texture features from the reference image to obtain a first texture feature.
[0050] Step 104, performing cross-attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain the second texture feature.
[0051] Exemplarily, for each frame image in the driving video, the second dynamic feature can be used as a query vector, the first dynamic feature can be used as a key vector, and the first texture vector can be used as a value vector to obtain the second texture feature through cross-attention calculation.
[0052] Since the human body posture information corresponding to the driving video is usually different from the human body posture information of the reference image, the misalignment of texture and structure information may cause inaccurate texture of the generated image. In this application, the texture features of the alignment of texture and motion structure can be obtained through the attention mechanism.
[0053] Step 105, splicing the second texture feature, the second dynamic feature and the noise to obtain a first splicing feature.
[0054] Exemplarily, the noise used may be Gaussian noise, and for each frame image in the driving video, the second texture feature, the second dynamic feature and the noise may be spliced together to obtain a first splicing feature corresponding to each frame image.
[0055] Step 106: Perform denoising processing according to the first splicing feature to generate a target video showing the target object.
[0056] Exemplarily, the first splicing features corresponding to each frame image in the driving video can be input into the denoising network, which is gradually denoised, and then the features output by the denoising network are decoded using the VAE decoder to generate a target video showing the target object. The target video shows the target object, and the action of the target object is consistent with the action of the driving object in the driving video. Thus, a video of the target object action is generated based on a given driving video of the driving human body action.
[0057] Exemplarily, the denoising network can be a U-Net network, which can include a convolution layer, a maximum pooling layer (downsampling), an attention layer, a deconvolution layer (upsampling), and a ReLU non-linear activation function.
[0058] In an embodiment of the present application, dynamic feature encoding is performed on a reference human body signal corresponding to a reference image and a driving human body signal corresponding to a driving video to obtain a first dynamic feature and a second dynamic feature, texture feature extraction is performed on the reference image to obtain a first texture feature, and cross-attention calculation is performed on the first dynamic feature, the second dynamic feature and the first texture feature, and a second texture feature in which the texture and the motion structure are aligned is obtained through an attention mechanism, thereby achieving effective alignment between the driving human body signal of the driving object and the reference image, thereby generating a target video based on the second texture feature combined with the second dynamic feature, thereby improving the accuracy of the texture in the generated video and improving the quality of the generated video.
[0059] Figure 2 A flowchart of a video generation method provided in another embodiment of the present application.
[0060] like Figure 2 As shown, the video generation method includes:
[0061] Step 201 : obtaining a reference human body signal corresponding to a reference image showing a target object and a driving human body signal corresponding to a driving video.
[0062] In the present application, step 201 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0063] Exemplarily, if the target object is the same as the driven object, key point detection can be performed on each frame image in the driving video to obtain the human body key points of the driving object, and three-dimensional face modeling and three-dimensional hand modeling can be performed based on each frame image in the driving video to obtain the first three-dimensional face model and three-dimensional hand model of the driving object, and the first three-dimensional face model and three-dimensional hand model are rendered respectively to obtain the first three-dimensional face rendering image and the three-dimensional hand rendering image, and then the driving human body signal is obtained according to the key points of the trunk and limbs in the human body key points, the first three-dimensional face rendering image and the three-dimensional hand rendering image. Among them, the driving human body signal can include the key points of the trunk and limbs of the driving object, the first three-dimensional face rendering image and the three-dimensional hand rendering image. The human body key points here can be 2D key points.
[0064] For example, the driving video is a video of the upper body movement of the driving object, and the key points of the torso and upper limbs of the driving object, the first three-dimensional face rendering image and the three-dimensional hand rendering image can be used as driving body signals.
[0065] Exemplarily, during the training phase, a frame of image can be extracted from the driving video as a reference image. In this case, the target object is the same as the driving object, and the key points of the torso and upper limbs of the driving object, the three-dimensional face rendering, and the three-dimensional hand rendering can be used as driving human body signals.
[0066] Since the 3D rendering can better represent the motion posture and occlusion relationship, it has a more accurate expression capability for complex gestures and frequent lip movements. Therefore, the 3D face rendering and 3D hand rendering of the driving object are used as the driving human body signal to generate the video of the driving target object, which can improve the accuracy of the human body motion of the target object in the generated video. In addition, using the key points of the human body trunk and limbs as the driving human body signal can reduce the amount of calculation while ensuring the accuracy of the trunk and limb movements, and improve the efficiency of video generation.
[0067] Exemplarily, if the target object is different from the driving object, then the first face shape coefficient of the target object can be obtained, and the second face shape coefficient of the driving object in the first three-dimensional face model can be replaced by the first face shape coefficient to obtain a second three-dimensional face model. The second three-dimensional face model is rendered to obtain a second three-dimensional face rendering image, and then the driving human body signal is obtained based on the key points of the torso and limbs of the driving object, the three-dimensional hand rendering image and the second three-dimensional face rendering image.
[0068] Among them, the three-dimensional face model may include a head posture coefficient, a face shape coefficient and a face expression coefficient. The face shape coefficient may refer to a characteristic coefficient used to describe the face shape in the three-dimensional face model. The face shape coefficient may be used to describe the overall shape and structure of the face.
[0069] For example, in the reasoning stage, the target object is different from the driving object, and the key points of the driving object's torso and limbs, the three-dimensional hand rendering, and the three-dimensional face rendering of the target object can be used as driving body signals.
[0070] Therefore, when the target object and the driving object are different, the face shape coefficient of the target object can be used to replace the face shape coefficient of the driving object in the driving video, so as to construct a driving human body signal according to the face shape coefficient of the target object and the driving video, so that the face shape in the generated video is the face shape of the target object, thereby meeting the video generation requirements when the target object and the driving object are different.
[0071] It should be noted that the above-mentioned limbs may refer to upper limbs or limbs, etc., which can be determined based on the limb conditions displayed in the driving video, and this application does not limit this.
[0072] Exemplarily, the first facial expression coefficient can be obtained from other videos or audios other than the driving video, and the first facial expression coefficient is used to replace the second facial expression coefficient of the driving object in the first three-dimensional face model to obtain a third three-dimensional face model, and the third three-dimensional face model is rendered to obtain a third three-dimensional face rendering image, and then the driving body signal is obtained according to the key points of the torso and limbs in the human body key points of the driving object, the three-dimensional hand rendering image and the third three-dimensional face rendering image. Here, the target object and the driving object can be the same or different, and there is no limitation on this.
[0073] The facial expression coefficient may refer to a characteristic coefficient used to describe facial expressions in a three-dimensional face model. The facial expression coefficient may be used to describe and simulate the movement of facial muscles and expression changes.
[0074] For example, video A is the driving video. Three-dimensional face modeling can be performed based on the image of the object contained in video B to obtain the three-dimensional face model of the object in video B, thereby obtaining the facial expression coefficient of the object in video B. The facial expression coefficient of the object in video B can be used to replace the facial expression coefficient in the three-dimensional face model of the driving object to obtain a new three-dimensional face model, and then a new three-dimensional face rendering image is obtained by rendering. The key points of the torso and limbs of the driving object, the three-dimensional hand rendering image of the driving object and the new three-dimensional face rendering image are used as driving human body signals.
[0075] Therefore, the facial expression coefficients from other videos or audios can be used to replace the facial expression coefficients of the driving object, so that the body movements, head postures, etc. of the target object in the generated video are consistent with those of the driving object. The expression of the target object is driven by other videos or audios, which can meet different video generation requirements.
[0076] Exemplarily, the reference human body signal can be obtained based on the key points of the torso and limbs of the target object in the reference image, the 3D face rendering of the target object, and the 3D hand rendering of the target object. For example, the key points of the torso and limbs of the target object, the 3D face rendering, and the 3D hand rendering can be used as the reference human body signal.
[0077] Step 202 , extracting dynamic features from the reference human body signal and the driving human body signal respectively to obtain a first dynamic feature and a second dynamic feature.
[0078] In the present application, step 202 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0079] Step 203: extract texture features from the reference image to obtain first texture features.
[0080] In the present application, step 203 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0081] Step 204: Calculate a first similarity matrix between the first dynamic feature and the second dynamic feature.
[0082] Exemplarily, the second dynamic feature may be used as a query, the first dynamic feature may be used as a key, and a dot product operation may be performed on the second dynamic feature and the first dynamic feature to obtain a first similarity matrix.
[0083] Step 205: Scaling and normalizing the first similarity matrix to obtain a first attention weight matrix.
[0084] In order to avoid the gradient vanishing or exploding problem caused by too large a dot product result during the calculation process, the first similarity matrix may be scaled, which may be achieved by dividing by a scaling factor, for example, the scaling factor may be the square root of the feature dimension.
[0085] After scaling the first similarity matrix, the scaled first similarity matrix can be normalized using a softmax function to ensure that the sum of the elements in each row is 1, thereby obtaining a first attention weight matrix.
[0086] Step 206: Perform weighted summation on the first texture features according to the first attention weight matrix to obtain second texture features.
[0087] Exemplarily, it can be considered that the first texture feature includes multiple vectors, each vector can be used as a value vector, and the first attention weight matrix can be used to perform weighted summation on each value vector in the first texture feature, thereby applying the attention weight to the value vector to obtain the second texture feature.
[0088] Step 207, concatenate the second texture feature, the second dynamic feature and the noise to obtain a first concatenated feature.
[0089] In the present application, step 207 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0090] Step 208: Perform denoising processing according to the first splicing feature to generate a target video showing the target object.
[0091] In the present application, step 208 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0092] In an embodiment of the present application, a first similarity matrix between the first dynamic feature and the second dynamic feature is calculated, and the first similarity matrix is scaled and normalized to obtain a first attention weight matrix. The first texture feature is weighted and summed through the first attention weight matrix, so that the attention to the first texture feature is dynamically adjusted according to the similarity between the second dynamic feature and the first dynamic feature. Attention is used on the first texture feature to achieve alignment between the texture of the reference image and the motion structure of the driving object, thereby improving the accuracy of human body movements in the generated video.
[0093] Figure 3 A flowchart of a video generation method provided in another embodiment of the present application.
[0094] like Figure 3 As shown, the video generation method includes:
[0095] Step 301 : obtaining a reference human body signal corresponding to a reference image showing a target object and a driving human body signal corresponding to a driving video.
[0096] In the present application, step 301 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0097] Step 302 , extracting dynamic features from the reference human body signal and the driving human body signal respectively to obtain a first dynamic feature and a second dynamic feature.
[0098] In the present application, step 302 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0099] Step 303: extract texture features from the reference image to obtain first texture features.
[0100] In the present application, step 303 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0101] Step 304, performing cross-attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain the second texture feature.
[0102] In the present application, step 304 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0103] Step 305: splice the second texture feature, the second dynamic feature and the noise to obtain a first spliced feature.
[0104] In the present application, step 305 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.
[0105] Step 306: splice the first dynamic feature and the first texture feature to obtain a second spliced feature.
[0106] Exemplarily, the first dynamic feature and the first texture feature corresponding to the reference image may be spliced to obtain a second spliced feature corresponding to the reference image, so that the spliced feature contains texture information and motion structure information of the target object.
[0107] Step 307: extract texture features from the second splicing features to obtain third texture features.
[0108] Exemplarily, the second splicing feature may be input into an appearance texture network, and the appearance texture network may further perform texture feature extraction.
[0109] Exemplarily, the appearance texture network may have the same structure as the denoising network, for example, both the appearance texture network and the denoising network may use a U-Net network.
[0110] Exemplarily, the appearance texture network may include an appearance attention layer, each appearance attention layer may perform self-attention calculation on the input features, so that third texture features at different scales may be obtained through multiple appearance attention layers.
[0111] Step 308: extract dynamic features from the first splicing features to obtain third dynamic features.
[0112] Exemplarily, the first concatenated feature obtained by concatenating the second dynamic feature, the second texture feature and the noise may be input into a denoising network and processed by the denoising network to obtain a third dynamic feature.
[0113] Exemplarily, the denoising network may include an appearance attention layer, and the first concatenated feature may be self-attentioned through the first appearance attention layer, and the subsequent appearance attention layer may perform self-attention on the feature inputted by the previous layer, so as to obtain a third dynamic feature outputted by each appearance attention layer.
[0114] Step 309: perform feature fusion based on the third texture feature and the third dynamic feature to obtain a first fused feature.
[0115] Exemplarily, the third texture feature may be fused with the third dynamic feature output by the corresponding layer in the denoising network to obtain a first fused feature.
[0116] Exemplarily, a second similarity matrix between the third texture feature and the third dynamic feature can be calculated, and then the second similarity matrix can be scaled and normalized to obtain a corresponding attention weight matrix, and then based on the attention weight matrix, the third texture feature can be weighted summed to obtain the first fusion feature.
[0117] In order to maximize the improvement effect of texture and structure alignment, when the third texture feature is fused with the third texture feature, the first similarity matrix of the first dynamic feature and the second dynamic feature can be used as a bias guide. Since the denoising network will perform downsampling, the scale of the third dynamic feature output by each layer is different from that of the first similarity matrix. Therefore, the first similarity matrix can be spatially scaled according to the size of the second similarity matrix to obtain a third similarity matrix, so that the scale of the third similarity matrix is consistent with the size of the second similarity matrix, and then the second similarity matrix is added to the third similarity matrix to obtain a fourth similarity matrix, and the fourth similarity matrix is scaled and normalized to obtain a second attention weight matrix, and then the third texture feature is weighted and summed according to the second attention weight matrix to obtain the first fused feature.
[0118] Therefore, the first similarity matrix between the first dynamic feature and the second dynamic feature can be used as a bias guide. In the cross-attention calculation of each appearance attention layer in the denoising network, the similarity matrix between the third texture feature and the third dynamic feature can be added to the similarity matrix obtained by spatially scaling the first similarity matrix, thereby strengthening the positions that need to be aligned, giving more attention, and improving the alignment effect of texture and motion structure.
[0119] Step 310: Generate a target video according to the first fusion feature.
[0120] Exemplarily, a hand texture feature library can be learned during the training phase, and the first fusion feature corresponding to each frame image in the driving video and the learned hand texture feature library can be input into the hand attention layer for cross-attention calculation to obtain the second fusion feature corresponding to each frame image. Then, the second fusion feature corresponding to each frame image is input into the temporal attention layer for temporal attention calculation to obtain the third fusion feature, and then the target video is generated based on the third fusion feature.
[0121] For example, the size of the hand texture feature library may be N*C, where N may represent the number of texture samples, and C may represent the feature dimension of each texture sample. It can be understood that the hand texture feature library includes N C-dimensional hand texture features.
[0122] Exemplarily, the temporal attention calculation may be a self-attention calculation based on the time dimension and channel dimension in the second fusion feature, which may capture the dependency relationship within the second fusion feature.
[0123] Exemplarily, the first fused feature can be used as a query, and the hand texture feature library can be used as a key and a value, and the second fused feature can be obtained through attention calculation.
[0124] Exemplarily, the third fused feature can be input into the next self-attention layer for self-attention, and then pass through the hand attention layer and the temporal attention layer for attention calculation in sequence until it passes through the last layer of the denoising network, and then the VAE decoder is used to decode the features finally output by the denoising network to generate the target video.
[0125] Therefore, the hand texture feature library obtained through learning is cross-attention calculated with the first fusion feature, so that the second fusion feature obtained can include the hand texture feature, enriching the feature representation information, and the temporal attention calculation is performed on the third fusion feature, which can capture the internal dependency of the third fusion feature, so as to generate a video based on the third fusion feature, which can ensure the temporal consistency of the generated video and the driving video, and reduce the blur and jitter of the generated video.
[0126] In order to enhance and restore the hand texture, the hand mask matrix can be used to enhance and restore the hand texture in a targeted manner. For example, the hand mask matrix can be obtained, and the first fusion feature and the hand texture feature library can be cross-attention calculated to obtain the first intermediate feature, and the hand mask matrix and the first intermediate feature can be masked to obtain the second intermediate feature, and then the second intermediate feature is added to the first fusion feature to obtain the second fusion feature.
[0127] The size of the hand mask matrix is the same as the first intermediate feature. For example, in the hand mask matrix, the element value corresponding to the hand area can be 1, and the element value corresponding to other parts of the human body except the hand can be 0. Then, when the hand mask matrix is multiplied by the first intermediate feature, the eigenvalue of the hand area remains unchanged, and the eigenvalues of all other areas are set to 0. The second intermediate feature obtained only contains the hand area feature and removes the interference of other non-hand areas.
[0128] Therefore, the hand texture can be enhanced through the hand mask matrix, and the enhanced restoration of the hand texture can be achieved. Therefore, for the complex gesture generation problem in human body driving, effective hand texture repair can be performed, which greatly improves the overall video generation quality.
[0129] Exemplarily, when performing cross-attention calculation on the first fusion feature and the hand texture feature library, the fifth similarity matrix between the first fusion feature and the hand texture feature library can be calculated, and the fifth similarity matrix can be scaled and normalized to obtain a third attention weight matrix, and then the hand texture feature library can be weighted summed according to the third weighted attention matrix to obtain the first intermediate feature.
[0130] Therefore, through the cross-attention mechanism, the deep interaction between the hand texture feature library and the first fusion feature can be captured, and the hand texture information can be focused on to improve the quality of the generated video.
[0131] In order to further improve the quality of the generated video, illustratively, the global features of the reference image can also be used as global information. For example, the CLIP (Contrastive Language-Image Pre-Training) model can be used to introduce global information. The CLIP model is a pre-training model. The CLIP model has two modalities, text modality and visual modality, which correspond to the text encoder and image encoder respectively.
[0132] Exemplarily, the image encoder in the CLIP model can be used to perform global feature extraction on the reference image to obtain global features, perform self-attention calculation on the second splicing feature to obtain a third intermediate feature, and perform cross-attention calculation on the third intermediate feature and the global feature to obtain a third texture feature, and then perform feature fusion based on the third texture feature and the third dynamic feature to obtain a first fusion feature, and generate a target video based on the first fusion feature. Therefore, when the texture feature is extracted from the second splicing feature, the global feature is introduced, so that the global feature is fused in the extracted third texture feature, thereby improving the quality of the generated video.
[0133] Exemplarily, the cross-attention calculation of the third intermediate feature and the global feature can be performed in the following manner: the sixth similarity matrix between the third intermediate feature and the global feature can be calculated, the sixth similarity matrix can be scaled and normalized to obtain a fourth attention weight matrix, and then the global feature can be weighted summed according to the fourth attention weight matrix to obtain the third texture feature.
[0134] Therefore, the global features of the target object are introduced, and the deep interaction between texture features and global features can be captured through the cross-attention mechanism, thereby improving the quality of the generated video.
[0135] Exemplarily, the first splicing feature can be input into the self-attention layer for self-attention calculation to obtain the fourth intermediate feature, and then the fourth intermediate feature is input into the multimodal attention layer for cross-attention calculation with the global feature of the reference image to obtain the third dynamic feature, and then feature fusion is performed based on the third texture feature and the third dynamic feature to obtain the first fused feature, and the target video is generated based on the first fused feature. Thus, when the dynamic feature is extracted from the first splicing feature, the global feature is introduced, so that the global feature is integrated into the extracted third dynamic feature, thereby improving the quality of the generated video.
[0136] Exemplarily, the cross-attention calculation of the fourth intermediate feature and the global feature can be performed in the following manner: the seventh similarity matrix between the fourth intermediate feature and the global feature can be calculated, the seventh similarity matrix can be scaled and normalized to obtain the fifth attention weight matrix, and then the global feature can be weighted summed according to the fifth attention weight matrix to obtain the third dynamic feature.
[0137] Therefore, the global features of the target object are introduced, and the deep interaction between dynamic features and global features can be captured through the cross-attention mechanism, thereby improving the quality of the generated video.
[0138] Exemplarily, the denoising network may include a self-attention layer, a multimodal attention layer, an appearance attention layer, a hand attention layer, and a temporal attention layer. The self-attention layer may perform self-attention calculation on the features inputted by the previous layer. The features outputted by the self-attention layer may be inputted into the multimodal attention layer of the next layer for cross-attention calculation with the global features of the reference image. The features outputted by the multimodal attention layer may be inputted into the appearance attention layer of the next layer for cross-attention calculation with the third texture features. The features outputted by the appearance attention layer may be inputted into the hand attention layer of the next layer for cross-attention calculation with the hand texture feature library. The features outputted by the hand attention layer may be inputted into the temporal attention layer of the next layer for temporal attention calculation. The features outputted by the last layer of the denoising network may be inputted into the VAE decoder for decoding to generate the target video.
[0139] In an embodiment of the present application, texture feature extraction can be performed on a second spliced feature obtained by splicing the first dynamic feature of the reference image and the first texture feature to obtain a third texture feature, dynamic feature extraction can be performed on the first spliced feature corresponding to the driving object to obtain a third dynamic feature, and then feature fusion can be performed on the third texture feature and the third dynamic feature, and a video can be generated based on the fused features obtained by the fusion. Thus, a video can be generated based on the texture features corresponding to the reference image and the dynamic features corresponding to the driving object, so that the generated video not only contains the texture information of the reference image, but also makes the action of the target object consistent with the action of the driving object, thereby improving the quality of the generated video.
[0140] In order to implement the above embodiment, the following Figure 4 To explain, Figure 4 A schematic diagram of a video generation process provided in an embodiment of the present application.
[0141] For human-driven videos with more complex motion patterns, an accurate human-driven signal can be proposed based on a combination of 2D and 3D solutions. Specifically, for coarse-grained motion parts such as upper limbs and torso, sparser 2D key points can be used to represent them; while for fine-grained motion parts such as face and hands, more accurate 3D face and 3D hand reconstruction renderings can be used to represent them. Compared with 2D key points, 3D renderings can better represent motion postures and occlusion relationships, and have more accurate expression capabilities for complex gestures and frequent lip movements.
[0142] The human-driven learning process can be divided into two parts: motion structure and appearance texture. Figure 4 As shown in the figure, this application is based on the diffusion model and adopts a dual-path U-Net network structure, in which the upper appearance texture network branch learns the appearance texture information based on randomly selected images, and the lower denoising network branch learns the motion structure information based on the input driving human body signal. The U-Net convolution structure used in the two branches is consistent, and the cross-attention mechanism is used for information exchange on the output features of the corresponding layers.
[0143] In this embodiment, the VAE of the pre-trained large model can be used as the local feature extractor and decoder in texture learning; at the same time, the dynamic feature encoder is used to characterize the motion structure information in the driving human body signal. In addition, the CLIP model can also be used to introduce global information into the diffusion model, such as Figure 4 As shown in , the CLIP encoder can be used to extract global features from the reference image to obtain the global feature f r c .
[0144] Since the human body posture of the driving human body signal is usually different from that of the reference image, the misalignment of texture and structural information will lead to inaccurate texture of the generated image. This application proposes a texture alignment technology with dynamic information enhancement based on the attention mechanism. Figure 4 As shown, the dynamic feature encoder is used to extract the second dynamic feature F for the input driving human body signal and the reference human body signal corresponding to the reference image. i and the first dynamic feature F r , the two dynamic feature sizes can be H*W*C. At the same time, the VAE encoder is used to extract the first texture feature Z of the reference image r By respectively i 、Fr and Z r As Query, Key and Value, the attention mechanism is used to calculate the second texture feature Z of structural alignment. i p , attention calculation see Figure 4 The specific calculation of texture alignment for dynamic information enhancement is shown in FIG. , and then the second texture feature Z i p With the second dynamic feature F i and Gaussian noise concatenation as the U-Net input of the denoising network.
[0145] Since the U-Net of the appearance texture and motion structure branches perform cross-attention mechanism calculations at different layers, in order to maximize the improvement effect of texture-structure alignment, this embodiment uses the second dynamic feature F i and the first dynamic feature F r The similarity matrix B i As a bias guide, a reference image bias attention mechanism is proposed. Specifically, in the cross attention calculation of each layer of the two U-Nets, the similarity matrix calculated by texture features and dynamic features must be compared with the similarity matrix of B. i The similarity matrices obtained by spatial scaling are added together, which can strengthen the positions that need to be aligned and give more attention.
[0146] To address the problem of hand texture synthesis, a hand restoration module is designed, which includes a learnable hand texture feature library with a feature size of N*C and a hand attention mechanism for optimizing hand texture. In the hand attention mechanism, the output feature map of any layer of the U-Net of the denoising network branch is used as the query, and the hand texture feature library is used as the key and value. In addition, a hand mask matrix can be used so that the calculation result of the hand attention is only added to the original output feature map in the hand area, and the texture of the hand is enhanced and restored in a targeted manner. The use of the hand mask matrix can also limit the information learned by the hand texture feature library to only be related to the hand, preventing the leakage of other redundant information.
[0147] like Figure 4 As shown in r With the first dynamic feature F r After splicing, it is input into the appearance texture network and the global feature f is introduced r c , both the appearance texture network and the denoising network contain self-attention layer and multimodal attention layer. The self-attention layer performs self-attention calculation on the features input from the previous layer, and the features output from the self-attention layer are input to the multimodal attention layer and the global feature f r cAfter performing cross attention calculation, the features output by the corresponding multimodal attention layers in the appearance texture network and the denoising network are input into the appearance attention layer for cross attention calculation. During the cross attention calculation of the appearance attention layer, the similarity matrix of the features output by the two multimodal attention layers can be compared with the similarity matrix B i The similarity matrices obtained by spatial scaling are added to enhance the positions that need to be aligned. The features output by the appearance attention layer and the learned hand texture feature library are input into the hand attention layer for cross-attention calculation. The features output by the hand attention layer are input into the temporal attention layer for temporal attention calculation. Finally, the features input by the denoising network are input into the VAE decoder for decoding to generate the video.
[0148] In the training phase of the diffusion model, after learning the appearance texture and motion structure at the image level, the parameters of the appearance texture and motion structure branch network structures can be fixed. In the denoising network branch, an additional temporal attention layer can be added separately to train the temporal modeling capability of the diffusion model. The training of the temporal attention layer can significantly reduce the blur and jitter of the video picture.
[0149] The video generation method of this application can rely on the generation framework of the two-way diffusion model to construct a driving human body signal suitable for precise human body driving, and propose a reference image bias attention mechanism to dynamically enhance texture alignment based on this driving human body signal to achieve effective alignment between the driving human body signal and the reference image. At the same time, based on the learnable hand texture feature library method and the cross-attention mechanism, we perform effective texture restoration for the complex gesture generation problem in human body driving, which greatly improves the overall generation quality.
[0150] In order to implement the above embodiment, the embodiment of the present application also proposes a video generating device. Figure 5 A schematic diagram of the structure of a video generating device provided in one embodiment of the present application.
[0151] like Figure 5 As shown, the video generating device 500 includes:
[0152] A first acquisition module 510 is used to acquire a reference human body signal corresponding to a reference image showing a target object and a driving human body signal corresponding to a driving video; wherein the reference human body signal is used to indicate a human body posture of the target object, and the driving human body signal is used to indicate a human body posture of the driving object in the driving video;
[0153] A dynamic feature extraction module 520 is used to extract dynamic features from the reference human body signal and the driving human body signal respectively to obtain a first dynamic feature and a second dynamic feature;
[0154] The texture feature extraction module 530 is used to extract texture features from the reference image to obtain a first texture feature;
[0155] A second acquisition module 540 is used to perform cross attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain a second texture feature;
[0156] A feature splicing module 550, configured to splice the second texture feature, the second dynamic feature and the noise to obtain a first splicing feature;
[0157] The generating module 560 is used to perform denoising processing according to the first splicing feature to generate a target video showing the target object.
[0158] Optionally, the second acquisition module 540 is used to:
[0159] Calculating a first similarity matrix between the first dynamic feature and the second dynamic feature;
[0160] Scaling and normalizing the first similarity matrix to obtain a first attention weight matrix;
[0161] According to the first attention weight matrix, the first texture features are weighted summed to obtain the second texture features.
[0162] Optionally, the generating module 560 is used to:
[0163] splicing the first dynamic feature and the first texture feature to obtain a second splicing feature;
[0164] Performing texture feature extraction on the second splicing feature to obtain a third texture feature;
[0165] Performing dynamic feature extraction on the first splicing feature to obtain a third dynamic feature;
[0166] Perform feature fusion according to the third texture feature and the third dynamic feature to obtain a first fused feature;
[0167] The target video is generated according to the first fusion feature.
[0168] Optionally, the generating module 560 is used to:
[0169] Obtaining a first similarity matrix between the first dynamic feature and the second dynamic feature;
[0170] Calculating a second similarity matrix between the third texture feature and the third dynamic feature;
[0171] Processing the first similarity matrix according to the size of the second similarity matrix to obtain a third similarity matrix; wherein the size of the third similarity matrix is the same as the size of the second similarity matrix;
[0172] Adding the third similarity matrix to the second similarity matrix to obtain a fourth similarity matrix;
[0173] Scaling and normalizing the fourth similarity matrix to obtain a second attention weight matrix;
[0174] According to the second attention weight matrix, weighted sum is performed on the third texture feature to obtain the first fusion feature.
[0175] Optionally, the generating module 560 is used to:
[0176] Performing cross-attention calculation based on the first fusion feature and the learned hand texture feature library to obtain a second fusion feature;
[0177] Performing temporal attention calculation on the second fused feature to obtain a third fused feature;
[0178] The target video is generated according to the third fusion feature.
[0179] Optionally, the generating module 560 is used to:
[0180] Get the hand mask matrix;
[0181] Performing cross attention calculation on the first fusion feature and the hand texture feature library to obtain a first intermediate feature;
[0182] Based on the hand mask matrix, masking the first intermediate feature to obtain a second intermediate feature;
[0183] The second intermediate feature is added to the first fused feature to obtain the second fused feature.
[0184] Optionally, the generating module 560 is used to:
[0185] Calculating a fifth similarity matrix between the first fusion feature and the hand texture feature library;
[0186] Scaling and normalizing the fifth similarity matrix to obtain a third attention weight matrix;
[0187] According to the third attention weight matrix, weighted sum is performed on the hand texture feature library to obtain the first intermediate feature.
[0188] Optionally, the generating module 560 is used to:
[0189] Performing global feature extraction on the reference image to obtain global features;
[0190] Performing self-attention calculation on the second concatenated feature to obtain a third intermediate feature;
[0191] A cross-attention calculation is performed on the third intermediate feature and the global feature to obtain the third texture feature.
[0192] Optionally, the generating module 560 is used to:
[0193] Calculating a sixth similarity matrix between the third intermediate feature and the global feature;
[0194] Scaling and normalizing the sixth similarity matrix to obtain a fourth attention weight matrix;
[0195] According to the fourth attention weight matrix, weighted sum is performed on the global features to obtain the third texture feature.
[0196] Optionally, the generating module 560 is used to:
[0197] Performing global feature extraction on the reference image to obtain global features;
[0198] Performing self-attention calculation on the first concatenated feature to obtain a fourth intermediate feature;
[0199] A cross-attention calculation is performed on the fourth intermediate feature and the global feature to obtain the third dynamic feature.
[0200] Optionally, the generating module 560 is used to:
[0201] Calculating a seventh similarity matrix between the fourth intermediate feature and the global feature;
[0202] Scaling and normalizing the seventh similarity matrix to obtain a fifth attention weight matrix;
[0203] According to the fifth attention weight matrix, the global features are weighted summed to obtain the third dynamic feature.
[0204] Optionally, the first acquisition module 510 is used to:
[0205] Performing key point detection on each frame image in the driving video to obtain the key points of the human body of the driving object;
[0206] Based on each frame image in the driving video, obtaining a first three-dimensional face model and a three-dimensional hand model of the driving object;
[0207] Rendering the first three-dimensional face model and the three-dimensional hand model to obtain a first three-dimensional face rendering image and a three-dimensional hand rendering image;
[0208] The driving human body signal is obtained according to the key points of the trunk and limbs among the key points of the human body, the first three-dimensional face rendering image and the three-dimensional hand rendering image.
[0209] Optionally, the first acquisition module is used to:
[0210] Performing key point detection on each frame image in the driving video to obtain the key points of the human body of the driving object;
[0211] Based on each frame image in the driving video, a first three-dimensional face model and a three-dimensional hand model of the driving object are obtained;
[0212] Acquire a first face shape coefficient of the target object, and use the first face shape coefficient to replace a second face shape coefficient of the driving object in the first three-dimensional face model to acquire a second three-dimensional face model;
[0213] Rendering the second three-dimensional face model and the three-dimensional hand model to obtain a second three-dimensional face rendering image and a three-dimensional hand rendering image;
[0214] The driving human body signal is obtained according to the key points of the trunk and limbs among the key points of the human body, the second three-dimensional face rendering image and the three-dimensional hand rendering image.
[0215] Optionally, the first acquisition module 510 is used to:
[0216] Performing key point detection on each frame image in the driving video to obtain the key points of the human body of the driving object;
[0217] Based on each frame image in the driving video, a first three-dimensional face model and a three-dimensional hand model of the driving object are obtained;
[0218] Obtaining a first facial expression coefficient, and using the first facial expression coefficient to replace a second facial expression coefficient of the driving object in the first three-dimensional face model, so as to obtain a third three-dimensional face model; wherein the first facial expression coefficient is obtained from other videos or audios except the driving video;
[0219] Rendering the third three-dimensional face model and the three-dimensional hand model to obtain a third three-dimensional face rendering image and a three-dimensional hand rendering image;
[0220] The driving human body signal is obtained according to the key points of the trunk and limbs among the key points of the human body, the third three-dimensional face rendering image and the three-dimensional hand rendering image.
[0221] It should be noted that the explanation of the aforementioned video generation method embodiment is also applicable to the video generation device of this embodiment, so it will not be repeated here.
[0222] In an embodiment of the present application, dynamic feature encoding is performed on a reference human body signal corresponding to a reference image and a driving human body signal corresponding to a driving video to obtain a first dynamic feature and a second dynamic feature, texture feature extraction is performed on the reference image to obtain a first texture feature, and cross-attention calculation is performed on the first dynamic feature, the second dynamic feature and the first texture feature, and a second texture feature in which the texture and the motion structure are aligned is obtained through an attention mechanism, thereby achieving effective alignment between the driving human body signal of the driving object and the reference image, thereby generating a target video based on the second texture feature combined with the second dynamic feature, thereby improving the accuracy of the texture in the generated video and improving the quality of the generated video.
[0223] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.
[0224] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0225] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 602 or a computer program loaded from a storage unit 608 to a RAM (Random Access Memory) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An I / O (Input / Output) interface 605 is also connected to the bus 604.
[0226] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0227] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the video generation method. For example, in some embodiments, the video generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the video generation method in any other appropriate manner (eg, by means of firmware).
[0228] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0229] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0230] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0231] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0232] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.
[0233] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0234] According to an embodiment of the present application, the present application also provides a computer program product, which, when an instruction processor in the computer program product executes, executes the video generation method proposed in the above embodiment of the present application.
[0235] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0236] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A video generation method, include: Acquire a reference human body signal corresponding to a reference image showing a target object and a driving human body signal corresponding to a driving video; wherein the reference human body signal is used to indicate a human body posture of the target object, and the driving human body signal is used to indicate a human body posture of the driving object in the driving video; Extracting dynamic features from the reference human body signal and the driving human body signal respectively to obtain a first dynamic feature and a second dynamic feature; Extracting texture features from the reference image to obtain first texture features; Performing cross attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain a second texture feature; splicing the second texture feature, the second dynamic feature and the noise to obtain a first splicing feature; De-noising is performed according to the first splicing feature to generate a target video showing the target object.
2. The method according to claim 1, in, The performing cross attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain the second texture feature includes: Calculating a first similarity matrix between the first dynamic feature and the second dynamic feature; Scaling and normalizing the first similarity matrix to obtain a first attention weight matrix; According to the first attention weight matrix, the first texture features are weighted summed to obtain the second texture features.
3. The method according to claim 1, in, The performing denoising processing according to the first splicing feature to generate a target video showing the target object includes: splicing the first dynamic feature and the first texture feature to obtain a second splicing feature; Performing texture feature extraction on the second splicing feature to obtain a third texture feature; Performing dynamic feature extraction on the first splicing feature to obtain a third dynamic feature; Perform feature fusion according to the third texture feature and the third dynamic feature to obtain a first fused feature; The target video is generated according to the first fusion feature.
4. The method according to claim 3, in, The step of fusing the third texture feature and the third dynamic feature to obtain a first fused feature includes: Obtaining a first similarity matrix between the first dynamic feature and the second dynamic feature; Calculating a second similarity matrix between the third texture feature and the third dynamic feature; Performing spatial scaling processing on the first similarity matrix according to the size of the second similarity matrix to obtain a third similarity matrix; wherein the size of the third similarity matrix is the same as the size of the second similarity matrix; Adding the third similarity matrix to the second similarity matrix to obtain a fourth similarity matrix; Scaling and normalizing the fourth similarity matrix to obtain a second attention weight matrix; According to the second attention weight matrix, weighted sum is performed on the third texture feature to obtain the first fusion feature.
5. The method according to claim 3, in, The step of generating the target video according to the first fusion feature includes: Performing cross-attention calculation based on the first fusion feature and the learned hand texture feature library to obtain a second fusion feature; Performing temporal attention calculation on the second fused feature to obtain a third fused feature; The target video is generated according to the third fusion feature.
6. The method according to claim 5, in, The performing cross attention calculation based on the first fusion feature and the learned hand texture feature library to obtain the second fusion feature includes: Get the hand mask matrix; Performing cross attention calculation on the first fusion feature and the hand texture feature library to obtain a first intermediate feature; Based on the hand mask matrix, masking the first intermediate feature to obtain a second intermediate feature; The second intermediate feature is added to the first fused feature to obtain the second fused feature.
7. The method according to claim 6, in, The performing cross attention calculation on the first fusion feature and the hand texture feature library to obtain a first intermediate feature includes: Calculating a fifth similarity matrix between the first fusion feature and the hand texture feature library; Scaling and normalizing the fifth similarity matrix to obtain a third attention weight matrix; According to the third attention weight matrix, weighted sum is performed on the hand texture feature library to obtain the first intermediate feature.
8. The method according to claim 3, in, The extracting texture features from the second splicing features to obtain third texture features includes: Performing global feature extraction on the reference image to obtain global features; Performing self-attention calculation on the second concatenated feature to obtain a third intermediate feature; A cross-attention calculation is performed on the third intermediate feature and the global feature to obtain the third texture feature.
9. The method according to claim 8, in, The performing cross attention calculation on the third intermediate feature and the global feature to obtain the third texture feature includes: Calculating a sixth similarity matrix between the third intermediate feature and the global feature; Scaling and normalizing the sixth similarity matrix to obtain a fourth attention weight matrix; According to the fourth attention weight matrix, weighted sum is performed on the global features to obtain the third texture feature.
10. The method according to claim 3, in, The step of extracting dynamic features from the first splicing features to obtain a third dynamic feature includes: Performing global feature extraction on the reference image to obtain global features; Performing self-attention calculation on the first concatenated feature to obtain a fourth intermediate feature; A cross-attention calculation is performed on the fourth intermediate feature and the global feature to obtain the third dynamic feature.
11. The method according to claim 10, in, The performing cross attention calculation on the fourth intermediate feature and the global feature to obtain the third dynamic feature includes: Calculating a seventh similarity matrix between the fourth intermediate feature and the global feature; Scaling and normalizing the seventh similarity matrix to obtain a fifth attention weight matrix; According to the fifth attention weight matrix, the global features are weighted summed to obtain the third dynamic feature.
12. The method according to any one of claims 1 to 11, in, The step of obtaining a driving human body signal corresponding to the driving video includes: Performing key point detection on each frame image in the driving video to obtain the key points of the human body of the driving object; Based on each frame image in the driving video, obtaining a first three-dimensional face model and a three-dimensional hand model of the driving object; Rendering the first three-dimensional face model and the three-dimensional hand model to obtain a first three-dimensional face rendering image and a three-dimensional hand rendering image; The driving human body signal is obtained according to the key points of the trunk and limbs among the key points of the human body, the first three-dimensional face rendering image and the three-dimensional hand rendering image.
13. The method according to any one of claims 1 to 11, in, The step of obtaining a driving human body signal corresponding to the driving video includes: Performing key point detection on each frame image in the driving video to obtain the key points of the human body of the driving object; Based on each frame image in the driving video, obtaining a first three-dimensional face model and a three-dimensional hand model of the driving object; Acquire a first face shape coefficient of the target object, and use the first face shape coefficient to replace a second face shape coefficient of the driving object in the first three-dimensional face model to acquire a second three-dimensional face model; Rendering the second three-dimensional face model and the three-dimensional hand model to obtain a second three-dimensional face rendering image and a three-dimensional hand rendering image; The driving human body signal is obtained according to the key points of the trunk and limbs among the key points of the human body, the second three-dimensional face rendering image and the three-dimensional hand rendering image.
14. The method according to any one of claims 1 to 11, in, The step of obtaining a driving human body signal corresponding to the driving video includes: Performing key point detection on each frame image in the driving video to obtain the key points of the human body of the driving object; Based on each frame image in the driving video, obtaining a first three-dimensional face model and a three-dimensional hand model of the driving object; Obtaining a first facial expression coefficient, and using the first facial expression coefficient to replace a second facial expression coefficient of the driving object in the first three-dimensional face model, so as to obtain a third three-dimensional face model; wherein the first facial expression coefficient is obtained from other videos or audios except the driving video; Rendering the third three-dimensional face model and the three-dimensional hand model to obtain a third three-dimensional face rendering image and a three-dimensional hand rendering image; The driving human body signal is obtained according to the key points of the trunk and limbs among the key points of the human body, the third three-dimensional face rendering image and the three-dimensional hand rendering image.
15. A video generating device, include: A first acquisition module is used to acquire a reference human body signal corresponding to a reference image showing a target object and a driving human body signal corresponding to a driving video; wherein the reference human body signal is used to indicate a human body posture of the target object, and the driving human body signal is used to indicate a human body posture of the driving object in the driving video; A dynamic feature extraction module, used to extract dynamic features from the reference human body signal and the driving human body signal respectively to obtain a first dynamic feature and a second dynamic feature; A texture feature extraction module, used to extract texture features from the reference image to obtain a first texture feature; A second acquisition module, configured to perform cross-attention calculation on the first dynamic feature, the second dynamic feature and the first texture feature to obtain a second texture feature; A feature splicing module, used for splicing the second texture feature, the second dynamic feature and the noise to obtain a first splicing feature; A generation module is used to perform denoising processing according to the first splicing feature to generate a target video showing the target object.
16. The device according to claim 15, in, The second acquisition module is used to: Calculating a first similarity matrix between the first dynamic feature and the second dynamic feature; Scaling and normalizing the first similarity matrix to obtain a first attention weight matrix; According to the first attention weight matrix, the first texture features are weighted summed to obtain the second texture features.
17. The device according to claim 15, in, The generating module is used for: splicing the first dynamic feature and the first texture feature to obtain a second splicing feature; Performing texture feature extraction on the second splicing feature to obtain a third texture feature; Performing dynamic feature extraction on the first splicing feature to obtain a third dynamic feature; Perform feature fusion according to the third texture feature and the third dynamic feature to obtain a first fused feature; The target video is generated according to the first fusion feature.
18. The device according to claim 17, in, The generating module is used for: Obtaining a first similarity matrix between the first dynamic feature and the second dynamic feature; Calculating a second similarity matrix between the third texture feature and the third dynamic feature; Processing the first similarity matrix according to the size of the second similarity matrix to obtain a third similarity matrix; wherein the size of the third similarity matrix is the same as the size of the second similarity matrix; Adding the first similarity matrix to the third similarity matrix to obtain a fourth similarity matrix; Scaling and normalizing the fourth similarity matrix to obtain a second attention weight matrix; According to the second attention weight matrix, weighted sum is performed on the third texture feature to obtain the first fusion feature.
19. The device according to claim 17, in, The generating module is used for: Performing cross-attention calculation based on the first fusion feature and the learned hand texture feature library to obtain a second fusion feature; Performing temporal attention calculation on the second fused feature to obtain a third fused feature; The target video is generated according to the third fusion feature.
20. The device according to claim 19, in, The generating module is used for: Get the hand mask matrix; Performing cross attention calculation on the first fusion feature and the hand texture feature library to obtain a first intermediate feature; Based on the hand mask matrix, masking the first intermediate feature to obtain a second intermediate feature; The second intermediate feature is added to the first fused feature to obtain the second fused feature.
21. The device according to claim 20, in, The generating module is used for: Calculating a fifth similarity matrix between the first fusion feature and the hand texture feature library; Scaling and normalizing the fifth similarity matrix to obtain a third attention weight matrix; According to the third attention weight matrix, weighted sum is performed on the hand texture feature library to obtain the first intermediate feature.
22. The device according to claim 17, in, The generating module is used for: Performing global feature extraction on the reference image to obtain global features; Performing self-attention calculation on the second concatenated feature to obtain a third intermediate feature; A cross-attention calculation is performed on the third intermediate feature and the global feature to obtain the third texture feature.
23. The device according to claim 22, in, The generating module is used for: Calculating a sixth similarity matrix between the third intermediate feature and the global feature; Scaling and normalizing the sixth similarity matrix to obtain a fourth attention weight matrix; According to the fourth attention weight matrix, weighted sum is performed on the global features to obtain the third texture feature.
24. The device of claim 17, in, The generating module is used for: Performing global feature extraction on the reference image to obtain global features; Performing self-attention calculation on the first concatenated feature to obtain a fourth intermediate feature; A cross-attention calculation is performed on the fourth intermediate feature and the global feature to obtain the third dynamic feature.
25. The device according to claim 24, in, The generating module is used for: Calculating a seventh similarity matrix between the fourth intermediate feature and the global feature; Scaling and normalizing the seventh similarity matrix to obtain a fifth attention weight matrix; According to the fifth attention weight matrix, the global features are weighted summed to obtain the third dynamic feature.
26. The device according to any one of claims 15 to 25, in, The first acquisition module is used to: Performing key point detection on each frame image in the driving video to obtain the key points of the human body of the driving object; Based on each frame image in the driving video, obtaining a first three-dimensional face model and a three-dimensional hand model of the driving object; Rendering the first three-dimensional face model and the three-dimensional hand model to obtain a first three-dimensional face rendering image and a three-dimensional hand rendering image; The driving human body signal is obtained according to the key points of the trunk and limbs among the key points of the human body, the first three-dimensional face rendering image and the three-dimensional hand rendering image.
27. The device according to any one of claims 15 to 25, in, The first acquisition module is used to: Performing key point detection on each frame image in the driving video to obtain the key points of the human body of the driving object; Based on each frame image in the driving video, obtaining a first three-dimensional face model and a three-dimensional hand model of the driving object; Acquire a first face shape coefficient of the target object, and use the first face shape coefficient to replace a second face shape coefficient of the driving object in the first three-dimensional face model to acquire a second three-dimensional face model; Rendering the second three-dimensional face model and the three-dimensional hand model to obtain a second three-dimensional face rendering image and a three-dimensional hand rendering image; The driving human body signal is obtained according to the key points of the trunk and limbs among the key points of the human body, the second three-dimensional face rendering image and the three-dimensional hand rendering image.
28. The device according to any one of claims 15 to 25, in, The first acquisition module is used to: Performing key point detection on each frame image in the driving video to obtain the key points of the human body of the driving object; Based on each frame image in the driving video, obtaining a first three-dimensional face model and a three-dimensional hand model of the driving object; Obtaining a first facial expression coefficient, and using the first facial expression coefficient to replace a second facial expression coefficient of the driving object in the first three-dimensional face model, so as to obtain a third three-dimensional face model; wherein the first facial expression coefficient is obtained from other videos or audios except the driving video; Rendering the third three-dimensional face model and the three-dimensional hand model to obtain a third three-dimensional face rendering image and a three-dimensional hand rendering image; The driving human body signal is obtained according to the key points of the trunk and limbs among the key points of the human body, the third three-dimensional face rendering image and the three-dimensional hand rendering image.
29. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.
30. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.
31. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.
Citation Information
Patent Citations
Video action migration deep learning system and method
CN114399708A
Video generation method and device, computer equipment and storage medium
CN114612595A
Face animation synthesis method and system and storage medium
CN116843806A
Video generation method and device and electronic equipment
CN117880606A
Virtual anchor whole-body video generation method and system based on diffusion model
CN117979115A
Cited By
Video generation method and device, and electronic equipment
CN119996784A
Video generation method and apparatus, and device and medium
WO2026152981A1
