Animation generation system and method based on video diffusion model and three-dimensional digital human-background mixed condition control

Abstract

The invention relates to an animation generation system and method based on a video diffusion model and three-dimensional digital human-background mixed condition control, and the system comprises a time sequence feature extraction module which extracts time sequence features from an avatar-back round video and an SMPL-X 3D grid video, and transmits the time sequence features to a mask embedding module and a video diffusion module; the mask embedding module converts the SMPL-X video to obtain a mask and transmits the mask to the video diffusion module; the audio driving module converts audio into an audio characteristic pattern and transmits the audio characteristic pattern to the video diffusion module; the multi-person layered mask module extracts facial features of each person from the mask of the face of each person and transmits the facial features to the video diffusion module; and the video diffusion module carries out motion expansion on the input reference image and outputs a complete video sequence. Compared with the prior art, the method has the advantages that high-fidelity and high-stability animation generation centered on human can be realized, and body-face cooperative control is realized.

Description

Technical Field

[0001] This invention relates to the field of image-to-video conversion technology, and in particular to an animation generation system and method based on a video diffusion model and three-dimensional digital human-background hybrid condition control. Background Technology

[0002] In recent years, video diffusion models have made significant progress in the field of image-to-video (I2V) conversion. By learning the mapping relationship between images and videos, video diffusion models enable the generation of dynamic videos from static images. However, when dealing with human-centered animation generation tasks, existing methods mainly rely on structural conditions (such as DWPose or SMPL-X) or explicit 3D methods. Structural condition methods (DWPose or SMPL-X) primarily utilize human pose estimation and 3D human models to guide animation generation. These methods perform well in standardized scenes, but are prone to distortion and instability in open domains and dynamic backgrounds. Explicit 3D methods generate animations by constructing detailed 3D models, achieving more stable driving effects, but their effectiveness is limited when handling non-rigid movements and complex backgrounds. Furthermore, in multi-person scenes, neither of these methods can provide fine-grained collaborative control over the entire body and faces of multiple individuals, resulting in unsatisfactory animation generation results.

[0003] In other words, existing technologies have the following drawbacks when dealing with open-domain portrait animation generation:

[0004] 1) Methods that rely on structural conditions have poor stability in dynamic backgrounds and complex postures, and are prone to distortion;

[0005] 2) Explicit 3D methods struggle to capture non-rigid motion and details;

[0006] 3) It is difficult to achieve precise facial expression and lip control as well as full-body control at the same time. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an animation generation system and method based on video diffusion model and three-dimensional digital human-background hybrid condition control, which can achieve more stable, realistic and controllable animation generation.

[0008] The objective of this invention can be achieved through the following technical solution: An animation generation system based on a video diffusion model and 3D digital human-background hybrid condition control, comprising a video diffusion module, a temporal feature extraction module, a mask embedding module, an audio driving module, and a multi-person layered mask module, wherein the temporal feature extraction module, the mask embedding module, the audio driving module, and the multi-person layered mask module are respectively connected to the input end of the video diffusion module, and the temporal feature extraction module is also connected to the input end of the mask embedding module. The temporal feature extraction module is used to extract temporal features from avatar-background video and SMPL-X 3D mesh video, and transmit the temporal features to the mask embedding module and the video diffusion module respectively.

[0009] The mask embedding module is used to convert SMPL-X video to obtain a mask and transmit the mask to the video diffusion module;

[0010] The audio driver module is used to convert audio into an audio feature map and transmit it to the video diffusion module;

[0011] The multi-person layered mask module is used to extract the facial features of each person from the mask of each person's face, and transmit the facial features and the corresponding facial mask to the video diffusion module;

[0012] The video diffusion module is used to perform motion expansion on the input reference image and output a complete video sequence.

[0013] Furthermore, the temporal feature extraction module includes two 3D convolutional units with the same architecture but different weights. The two 3D convolutional units are used to embed the avatar-background video and the SMPL-X 3D mesh video into the feature dimension, respectively, to obtain the corresponding temporal feature map.

[0014] Furthermore, the 3D convolutional unit includes five 3D convolutional layers, each of which includes one 3D convolution and one ReLU activation function layer.

[0015] Furthermore, the operation of the mask embedding module includes:

[0016] Each frame of the SMPL-X video is binarized into a 0-1 mask, where the areas with values ​​are 1 (i.e., the background is set to 1) and the black areas are 0 (i.e., the human body area is set to 0). Then, Gaussian filtering is used to smooth the edges of the mask, thus obtaining the final masked video.

[0017] Furthermore, the audio driver module includes a pre-trained wav2vec model and a feature mapping unit with 5 layers of MLP (Multilayer Perceptron).

[0018] Furthermore, the working process of the audio driving module includes: inputting audio into a pre-trained wav2vec model to obtain a series of feature maps, and then passing these feature maps through a feature mapping unit to obtain the final audio feature map.

[0019] Furthermore, the multi-person layered mask module includes a feature extractor for extracting facial features.

[0020] Furthermore, the operation of the multi-person layered mask module includes:

[0021] First, each person's facial image is extracted from their facial mask. Then, facial features are extracted using a feature extractor. Finally, the extracted facial features are combined with the person's facial mask to produce the final output.

[0022] Furthermore, the operation of the video diffusion module includes:

[0023] First, the reference image is embedded into the feature dimension. Then, it is fed into the diffusion model architecture along with temporal features, masked video, audio feature map, facial features and corresponding facial mask. A LoRA (low-rank adaptation) layer is added to the diffusion model architecture for parameter tuning. Only the LoRA layer is trained, while the other backbone parameters are frozen. Finally, the generated animation is output, which contains the complete video sequence.

[0024] An animation generation method based on a video diffusion model and 3D digital human-background hybrid condition control includes the following steps:

[0025] Extracting temporal features from avatar-background videos and SMPL-X 3D mesh videos;

[0026] A spatial variation masking strategy is adopted to binarize each frame of the SMPL-X video into a 0-1 mask. The mask is then downsampled and spatially Gaussian blurred to obtain a flexible boundary, thus generating a masked video.

[0027] The audio is converted into a feature vector using the wav2vec technology. Based on MLP, the audio feature vector is then converted into task-wide driving features, which is the audio feature map.

[0028] Each person's facial image is extracted from their facial mask, facial features are extracted from them, and each person's facial features are matched with their corresponding facial mask.

[0029] The input reference image is embedded into the feature dimension, and then imported into the diffusion model architecture along with temporal features, masked video, audio feature maps, each person's facial features and corresponding facial masks. This diffusion model architecture adds a LoRA (low-rank adaptation) layer to the DiT (DiffusionTransformer) network for parameter tuning. Only the LoRA layer is trained, while the other backbone parameters are kept frozen. The final output is the generated animation, which contains the complete video sequence.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] This invention connects a temporal feature extraction module, a mask embedding module, an audio driving module, and a multi-person layered mask module to a video diffusion module. The temporal feature extraction module extracts temporal features from avatar-background videos and SMPL-X 3D mesh videos; the mask embedding module converts SMPL-X videos to obtain masks; the audio driving module converts audio into audio feature maps; the multi-person layered mask module extracts facial features from each person's face mask and combines them with the corresponding facial mask; and the video diffusion module performs motion expansion on the input reference image to output a complete video sequence. This introduces a "3D digital human-background" conditional mechanism, modeling the animation generation task as a video restoration task. By using the character and background as conditional inputs, the diffusion model is guided to generate an animation sequence consistent with the input image, achieving high-fidelity and stable animation generation. Furthermore, based on the "3D digital human-background" conditional mechanism, an audio-controlled facial expression and lip control mechanism is introduced, enabling multi-person full-body and multi-face multi-granular collaborative control.

[0032] In this invention, the temporal feature module is constructed based on a 3D convolutional network. Its inputs are SMPL-X mesh animation and avatar video. By extracting the temporal structure information from the avatar-background video and the SMPL-X 3D mesh video, it is beneficial to guide the spatial consistency and motion control in the subsequent video diffusion module during the generation process.

[0033] In this invention, the mask embedding module uses a spatial variation mask and combines SMPL-X video structure information for region recognition. The human body region is set to 0 (needs refinement) and the background is set to 1 (should be retained). This allows the "generated region" and "retained region" in the diffusion process to be specified in a structure-aware manner. This facilitates efficient region-level control and refinement, enabling the video diffusion module to focus on high-quality generation of the human body when generating animation.

[0034] This invention utilizes an audio driver module and a multi-person layered mask module to obtain audio feature maps and each person's facial features and corresponding facial masks, which facilitates the subsequent implementation of facial expression and lip control for audio control, enabling multi-person full-body and facial multi-granular collaborative control.

[0035] In the video diffusion module, this invention adds a LoRA (low-rank adaptation) layer to each DiT (Diffusion Transformer) network for parameter tuning. Only the LoRA layer is trained, while the other backbone parameters are frozen. This ensures that the model inherits prior knowledge and can generate complete video sequences that meet semantic conditions. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0037] Figure 2 This is a schematic diagram of the method flow of the present invention;

[0038] The markings in the diagram are as follows: 1. Video diffusion module, 2. Temporal feature extraction module, 3. Mask embedding module, 4. Audio driving module, 5. Multi-person layered mask module. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0040] Example

[0041] like Figure 1 As shown, an animation generation system based on a video diffusion model and 3D digital human-background hybrid condition control includes a video diffusion module 1, a temporal feature extraction module 2, a mask embedding module 3, an audio driving module 4, and a multi-person layered mask module 5. The temporal feature extraction module 2, the mask embedding module 3, the audio driving module 4, and the multi-person layered mask module 5 are respectively connected to the input end of the video diffusion module 1. The temporal feature extraction module 2 is also connected to the input end of the mask embedding module 3. The temporal feature extraction module 2 is used to extract temporal features from avatar-background video and SMPL-X 3D mesh video, and transmit the temporal features to the mask embedding module 3 and the video diffusion module 1 respectively.

[0042] The mask embedding module 3 is used to convert SMPL-X video to obtain a mask and transmit the mask to the video diffusion module 1;

[0043] The audio driver module 4 is used to convert audio into an audio feature map and transmit it to the video diffusion module 1;

[0044] The multi-person layered mask module 5 is used to extract the facial features of each person from the mask of each person's face, and transmit the facial features and the corresponding facial mask to the video diffusion module 1;

[0045] The video diffusion module 1 is used to perform motion expansion on the input reference image and output a complete video sequence.

[0046] Based on the above system, an animation generation method based on a video diffusion model and 3D digital human-background hybrid condition control is implemented, such as... Figure 2 As shown, it includes the following steps:

[0047] Extracting temporal features from avatar-background videos and SMPL-X 3D mesh videos;

[0048] A spatial variation masking strategy is adopted to binarize each frame of the SMPL-X video into a 0-1 mask. The mask is then downsampled and spatially Gaussian blurred to obtain a flexible boundary, thus generating a masked video.

[0049] The audio is converted into a feature vector using the wav2vec technology. Based on MLP, the audio feature vector is then converted into task-wide driving features, which is the audio feature map.

[0050] Each person's facial image is extracted from their facial mask, facial features are extracted from them, and each person's facial features are matched with their corresponding facial mask.

[0051] The input reference image is embedded into the feature dimension, and then imported into the diffusion model architecture along with temporal features, masked video, audio feature maps, each person's facial features and corresponding facial masks. This diffusion model architecture adds a LoRA (low-rank adaptation) layer to the DiT (DiffusionTransformer) network for parameter tuning. Only the LoRA layer is trained, while the other backbone parameters are kept frozen. The final output is the generated animation, which contains the complete video sequence.

[0052] This embodiment applies the above scheme to build a video diffusion module 1, a temporal feature extraction module 2, a mask embedding module 3, an audio driving module 4, and a multi-person layered mask module 5, respectively. Specifically:

[0053] I. Video Distribution Module

[0054] Using a video diffusion model as its core, the model extends the motion of a reference image through conditional guidance to generate a complete video sequence that meets semantic conditions. The generation target is changed from "target video" to "reference image + target video," enhancing the model's initial frame consistency and training stability. A LoRA (low-rank adaptation) layer is added to each DiT (Diffusion Transformer) module for parameter tuning; only the LoRA layer is trained, while the remaining backbone parameters are frozen to ensure the model inherits prior knowledge.

[0055] The inputs to the video diffusion module are: a reference image, the output features of the temporal feature extraction module, the output features of the mask embedding module, the output features of the audio driving module, and the output features of the multi-person layered mask module.

[0056] The output of the video diffusion module is the final generated video.

[0057] The intermediate data processing of the video diffusion module is as follows: First, the reference image is embedded into the feature dimension. Then, it is imported into the diffusion model architecture along with the output features of the temporal feature extraction module, the mask embedding module, the audio driving module, and the multi-person layered mask module. A LoRA (low-rank adaptation) layer is added to the module for parameter tuning. Only the LoRA layer is trained, and the other backbone parameters are kept frozen to ensure that the model inherits prior knowledge. Finally, the generated video is output.

[0058] II. Temporal Feature Extraction Module

[0059] Temporal structure information is extracted from avatar-background videos and SMPL-X 3D mesh videos to guide spatial consistency and motion control during the generation process. A 3D convolutional network (3D Conv) is constructed, with SMPL-X mesh animation and avatar videos as input. The SMPL-X video input is a structured and motion-clean mesh animation, providing geometric constraints and region guidance; the output temporal features are used by the mask embedding module and the diffusion network.

[0060] The input to the temporal feature extraction module is: avatar-background video and SMPL-X 3D mesh video.

[0061] The output of the temporal feature extraction module is: temporal features.

[0062] The intermediate data processing of the temporal feature extraction module is as follows: using two 3D convolutional modules with the same architecture but different weights, the avatar-background video and the SMPL-X 3D mesh video are embedded into the feature dimension to obtain the corresponding feature maps. Each 3D convolutional module includes 5 3D convolutional layers, and each 3D convolutional layer includes a 3D convolution and a ReLU activation function layer.

[0063] III. Mask Embedding Module

[0064] This approach uses a structure-aware method to specify the "generated region" and "retained region" during the diffusion process, achieving efficient region-level control and refinement. The mask is a binary tensor, where 1 indicates retention (e.g., the input image) and 0 indicates the frame to be generated. This scheme introduces a spatially variable mask, combined with SMPL-X structural information for region recognition, setting the human body region to 0 (requiring refinement) and the background to 1 (to be retained). The mask is downsampled and spatially Gaussian blurred to obtain flexible boundaries, generating the final mask. This mask is input into the diffusion network, enabling the network to focus on high-quality generation of the human body.

[0065] The input to the mask embedding module is: SMPL-X 3D mesh video.

[0066] The output of the mask embedding module is: mask video.

[0067] The intermediate data processing of the mask embedding module is as follows: First, each frame of the SMPL-X video is binarized into a 0-1 mask (values ​​are 1, and black values ​​are 0). Then, Gaussian filtering is used to smooth the edges of the mask, thus obtaining the final masked video. Smoothing allows the mask to flexibly handle situations where clothing is relatively loose, i.e., the actual edges of the human body do not match the edges of the SMPLX mask.

[0068] IV. Audio Driver Module

[0069] The audio is converted into feature vectors using the wav2vec technology. The audio features are then converted into task-wide driving features by the MLP-based feature mapping module, and then imported into the DiT backbone network.

[0070] The input to the audio driver module is the audio being driven.

[0071] The output of the audio driver module is: a feature map of audio embedding.

[0072] The intermediate data processing of the audio driver module is as follows: First, the audio is input into the pre-trained wav2vec module to obtain a series of feature maps. Then, these feature maps are passed through a 5-layer trainable multilayer perceptron (MLP) to obtain the final audio feature map.

[0073] V. Multi-person layered mask module

[0074] Multiple facial masks and corresponding ID embeddings enable multi-person body-face multi-granularity collaborative control. A feature extractor imports the ID embedding features corresponding to the facial masks into the backbone DiT (DiffusionTransformer) network, thus achieving multi-person control.

[0075] The input to the multi-person layered mask module is: the mask of each person's face.

[0076] The output of the multi-person layered mask module is: the feature ID Embedding of each person.

[0077] The intermediate data processing of the multi-person layered mask module is as follows: First, each person's facial image is extracted using their facial mask, and then facial features are extracted through a series of feature extraction networks. The extracted facial features are then combined with the person's mask as the final output. This strategy enables the model to control the speech of a specific individual.

[0078] To verify the effectiveness of this approach, this embodiment compares it with other existing methods on two datasets (human dance dataset and HumanVid dataset):

[0079] 1. On the human dance dataset, compared with the second-best method, our approach improved SSIM (Structural Similarity Index Measure) by 0.014, PSNR (Peak Signal-to-Noise Ratio) by 0.839 dB, LPIPS (Learned Perceptual Image Patch Similarity) (lower is better) by 0.0167, FID (Fréchet Inception Distance) (lower is better) by 4.244, and FVD (Fréchet Video Distance) (lower is better) by 52.01.

[0080] 2. On the HumanVid dataset, compared with the second-best method, this approach improves SSIM by 0.1761, PSNR by 5.952 dB, reduces LPIPS (lower is better) by 0.1301, FID (lower is better) by 52.018, and FVD (lower is better) by 410.97.

[0081] The experimental results above show that the proposed method outperforms existing methods in terms of stability, realism, and detail capture, and achieves significant improvements in multiple metrics.

[0082] In summary, this solution introduces a "3D digital human-background" conditional mechanism and an audio-controlled facial expression and lip control mechanism to achieve multi-person, full-body, and multi-grained collaborative control of the face. It models open-domain human portrait animation as a restoration task, thereby achieving more stable, realistic, and controllable human-centered animation generation. Under dynamic backgrounds and complex postures, this solution generates more stable animations with less distortion; it can capture more details, resulting in more realistic animations; and it can simultaneously achieve body-face collaborative control.

Claims

1. An animation generation system based on video diffusion model and three-dimensional digital human-background mixed condition control, characterized in that, The application relates to a video diffusion method, which comprises a video diffusion module (1), a time sequence feature extraction module (2), a mask embedding module (3), an audio driving module (4) and a multi-person layered mask module (5), wherein the time sequence feature extraction module (2), the mask embedding module (3), the audio driving module (4) and the multi-person layered mask module (5) are connected with the input end of the video diffusion module (1), the time sequence feature extraction module (2) is also connected with the input end of the mask embedding module (3), the time sequence feature extraction module (2) is used for extracting time sequence features from avatar-background videos and SMPL-X 3D grid videos and transmitting the time sequence features to the mask embedding module (3) and the video diffusion module (1) respectively. The mask embedding module (3) is used for converting SMPL-X videos to obtain masks and transmitting the masks to the video diffusion module (1). The audio driving module (4) is used for converting audio into an audio feature map and transmitting the audio feature map to the video diffusion module (1). The multi-person layered mask module (5) is used for extracting facial features of each person from the mask of the face of each person and transmitting the facial features and the corresponding face mask to the video diffusion module (1). The video diffusion module (1) is used for performing motion expansion on a reference image to output a complete video sequence.

2. The animation generation system based on video diffusion model and three-dimensional digital human-background mixed condition control according to claim 1, wherein, The time sequence feature extraction module (2) comprises two 3D convolution units with the same architecture but different weights, and the two 3D convolution units are used for embedding the avatar-background videos and the SMPL-X 3D grid videos into feature dimensions to obtain corresponding time sequence feature maps.

3. The animation generation system based on video diffusion model and three-dimensional digital human-background mixed condition control according to claim 2, characterized in that, The 3D convolution unit comprises five 3D convolution layers, each of which comprises a 3D convolution and a ReLU activation function layer.

4. The animation generation system based on video diffusion model and three-dimensional digital human-background mixed condition control according to claim 1, characterized in that, The working process of the mask embedding module (3) comprises the following steps: Each frame of the SMPL-X video is binarized into a 0-1 mask, wherein the value is 1, that is, the background is set to 1, and the black part is 0, that is, the human body region is set to 0, and then the edges of the mask are smoothed by using a Gaussian filter to obtain a final mask video.

5. The animation generation system based on video diffusion model and three-dimensional digital human-background mixed condition control according to claim 1, wherein, The audio driving module (4) comprises a pre-trained wav2vec model and a feature mapping unit provided with five layers of MLP.

6. The animation generation system based on video diffusion model and three-dimensional digital human-background mixed condition control according to claim 5, characterized in that, The working process of the audio driving module (4) comprises the following steps: inputting audio into the pre-trained wav2vec model to obtain a series of feature maps, and then passing the feature maps through the feature mapping unit to obtain a final audio feature map.

7. The animation generation system based on video diffusion model and three-dimensional digital human-background mixed condition control according to claim 1, characterized in that, The multi-person layered mask module (5) comprises a feature extractor used for extracting facial features.

8. The animation generation system based on video diffusion model and three-dimensional digital human-background mixed condition control according to claim 7, characterized in that, The working process of the multi-person layered mask module (5) comprises the following steps: First, the face picture of each person is intercepted from the face mask of each person, and then the facial features are extracted through the feature extractor, and finally the extracted facial features of each person are combined with the face mask of the person to serve as the final output.

9. The animation generation system based on video diffusion model and three-dimensional digital human-background mixed condition control according to claim 1, wherein, The working process of the video diffusion module (1) comprises the following steps: Firstly, the reference image is embedded into the feature dimension, and then input into the diffusion model architecture together with the temporal features, the mask video, the audio feature map, the face features and the corresponding face masks. A LoRA layer is added to the diffusion model architecture for parameter regulation. Only the LoRA layer is trained, and the remaining backbone parameters are kept frozen. Finally, the generated animation containing the complete video sequence is output.

10. An animation generation method based on a video diffusion model and a three-dimensional digital human-background mixed condition control, characterized in that, The method comprises the following steps: Temporal features are extracted from the avatar-background video and the SMPL-X 3D mesh video; A spatial variation mask strategy is adopted to binarize each frame of the SMPL-X video into a 0-1 mask. The mask is down-sampled and processed with spatial Gaussian blur to obtain a flexible boundary, and a mask video is generated; Wav2vec technology is adopted to convert the audio into a feature vector. Based on the MLP, the audio feature vector is converted into the task's pervasive driving feature, i.e., an audio feature map is obtained; The face pictures of each person are cut out from the face masks of each person, the face features are extracted therefrom, and the face features of each person are combined with the corresponding face masks of each person; The input reference image is embedded into the feature dimension, and then input into the diffusion model architecture together with the temporal features, the mask video, the audio feature map, the face features of each person and the corresponding face masks. A LoRA layer is added to the diffusion model architecture in the DiT network for parameter regulation. Only the LoRA layer is trained, and the remaining backbone parameters are kept frozen. Finally, the generated animation containing the complete video sequence is output.

Images