Dynamic video generation method and system based on diffusion model

Abstract

The invention discloses a dynamic video generation method and system based on a diffusion model, and belongs to the technical field of image processing, and the method comprises the steps: obtaining a plurality of role pictures and video clips to construct a data set; constructing an image dynamic generation model based on the diffusion model; extracting a posture sequence image of a role in the data set, rendering a posture sequence to obtain a posture skeleton sequence image, and inputting the data set and the posture skeleton sequence image into an image dynamic generation model for training to obtain a trained image dynamic generation model; and obtaining a static image to be generated and inputting the static image into the trained image dynamic generation model to obtain a dynamic video corresponding to the static image. According to the method, the stability of the time dimension in the dynamic video generation process and the consistency of video textures are ensured through the improved diffusion model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to a method and system for generating dynamic video based on a diffusion model. Background Art

[0002] In the field of image generation, diffusion model-based methods have dominated due to their excellent generation effects, significantly promoting the development of this field. In order to alleviate the computational burden, the latent diffusion model came into being, which cleverly balanced efficiency and performance by performing denoising in the latent space. Innovative research such as Control Net and T2I-adapter deepened the exploration of the controllability of visual generation by introducing additional coding layers, making controllable generation under different conditions (posture, mask, edge, depth, etc.) possible.

[0003] With the rapid development of text-to-image generation technology, research on text-to-video has followed closely, and has widely borrowed the model architecture of the former. Many studies focus on strengthening the inter-frame attention mechanism, taking the text-to-image (T2I) model as the cornerstone and expanding it to the field of video generation. Some work cleverly transforms the pre-trained T2I model into a video generation engine by incorporating the time dimension.

[0004] In addition, the expansion of image-to-video capabilities is also eye-catching, especially image animation technology, which aims to dynamically generate continuous images or videos based on one or more input images. In the latest progress of human image animation, diffusion models have gradually become a key driving force with their excellent generation quality and stable control capabilities. For example, PIDM incorporates specific textures into the pose transfer process through texture diffusion blocks, LFDM constructs optical flow sequences in latent space to achieve condition-driven image deformation, and LEO creatively expresses motion as a sequence of flowcharts and synthesizes motion codes using diffusion models. DreamPose uses pre-trained stable diffusion models and adapters to cleverly integrate image embedding of CLIP and VAE. The DisCo project draws inspiration from Control Net to decouple pose and background control. Although these methods have made significant progress in improving generation quality and controllability, the field of human video generation still faces many challenges, such as texture consistency and stability in the temporal dimension.

[0005] Therefore, how to provide a method and system for generating dynamic videos with consistent texture and stable temporal dimension is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0006] In view of this, the present invention provides a method and system for generating dynamic videos based on a diffusion model. Based on the improved diffusion model, the stability of the time dimension and the consistency of the video texture in the dynamic video generation process are ensured.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A method for generating dynamic video based on a diffusion model, comprising:

[0009] Obtain several character images and video clips to construct a data set;

[0010] Construct an image dynamic generation model based on the diffusion model;

[0011] Extracting posture sequence images of the characters in the data set, rendering the posture sequence images to obtain posture skeleton sequence images, and inputting the data set and the posture skeleton sequence images into the image dynamic generation model for training to obtain a trained image dynamic generation model;

[0012] The static image to be generated is obtained and input into the trained image dynamic generation model. The trained image dynamic generation model obtains the character reference image features according to the static image, and generates a dynamic video according to the character reference image features and posture skeleton sequence images.

[0013] Preferably, obtaining the character reference image features according to the static image includes:

[0014] Extracting detail features of the static image for spatial attention;

[0015] Extracting semantic features of the static image through an image encoder for cross attention;

[0016] The detail features and the semantic features are fused to generate character reference graph features.

[0017] Preferably, generating a dynamic video according to the character reference image features and the posture skeleton sequence images includes:

[0018] Extracting detailed features of the character reference image features and using them for spatial attention;

[0019] Extracting semantic features of the character reference graph features for cross attention;

[0020] Determining the temporal relationship of actions for temporal attention according to the posture skeleton sequence images;

[0021] The output of combining spatial attention, cross attention, and temporal attention is decoded into dynamic video clips.

[0022] Preferably, the posture sequence of the character in the data set is extracted, and the posture sequence is rendered to obtain the posture skeleton sequence image, including: using a convolution layer to align the posture sequence image of the video clip in the data set with the noise at the same resolution, and adding the processed posture sequence image to the noise hidden layer to obtain the posture skeleton sequence image.

[0023] On the other hand, the present invention provides a dynamic video generation system based on a diffusion model, comprising:

[0024] The acquisition module is used to obtain several character pictures and video clips to form a data set;

[0025] Construction module, constructs an image dynamic generation model based on the diffusion model;

[0026] A sequence generation module extracts posture sequence images of the characters in the data set, and renders the posture sequence images to obtain posture skeleton sequence images;

[0027] A training module, inputting the data set and the posture skeleton sequence image into the image dynamic generation model for training to obtain a trained image dynamic generation model;

[0028] The generation module obtains the static image to be generated and inputs it into the trained image dynamic generation model. The trained image dynamic generation model obtains the character reference image features according to the static image, and generates a dynamic video according to the character reference image features and the posture skeleton sequence image.

[0029] Preferably, the generating module comprises:

[0030] A reference image feature extraction unit, used to obtain a character reference image feature according to the static image;

[0031] A denoising unit is used to generate a dynamic video based on the character reference image features and posture skeleton sequence images.

[0032] Preferably, the reference image feature extraction unit comprises:

[0033] A feature extraction layer, used to extract detail features of the static image for spatial attention;

[0034] An image encoder, for extracting semantic features of the static image for cross attention;

[0035] A fusion unit is used to fuse the detail feature and the semantic feature to generate a character reference graph feature.

[0036] Preferably, the denoising unit comprises:

[0037] A spatial attention layer, used to extract detailed features of the character reference image features and use them for spatial attention;

[0038] A cross attention layer, used for extracting semantic features of the character reference graph features for cross attention;

[0039] A temporal attention layer, for determining a temporal relationship of actions for temporal attention according to the sequence of posture skeleton images;

[0040] The video decoder decodes the output results of the spatial attention layer, the cross attention layer, and the temporal attention layer into a dynamic video clip.

[0041] Preferably, the sequence generation module includes:

[0042] A convolutional layer, used to align the posture sequence images of the video clips in the data set with the noise at the same resolution;

[0043] The noise hidden layer is used to obtain a posture skeleton sequence image according to the processed posture sequence image.

[0044] It can be seen from the above technical solutions that compared with the prior art, the present invention discloses a method and system for generating dynamic videos based on a diffusion model. First, the posture sequence is encoded and then fused with multi-frame noise. Then, the denoising network is used to denoise and generate the video. In the denoising process, spatial attention, cross attention and temporal attention are introduced to ensure the continuity of the character movements in the generated video. Even in the case of intense exercise, it can maintain temporal consistency with the reference image and show temporal continuity between frames. Furthermore, the present invention adds a reference image feature extraction network, so that the denoising network selectively learns related features in the same feature space from the reference image feature extraction network, improves the detail features in the video generation process, and can generate high-definition and realistic character details. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A schematic diagram of the overall framework provided by the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The embodiment of the present invention discloses a method for generating dynamic video based on a diffusion model, comprising:

[0049] Several character images and video clips are obtained to construct a data set; the video clips are generally 2-10 seconds long.

[0050] The image dynamic generation model is constructed based on the diffusion model; the denoising network in the constructed image dynamic generation model is configured based on the design of Stable Diffusion (SD), and the initial input of the network consists of multiple frames of noise. The model contains three key components: ReferenceNet: encoding the appearance features of the characters in the reference image, in which the self-attention layer is replaced by the space-attention layer; Pose Guider: encoding motion control signals to achieve controllable actions of the character; Temporal Layer: encoding time relationships to ensure the continuity of the character's movement.

[0051] Extract the pose sequence images of the characters in the dataset, render the pose sequence to obtain the pose skeleton sequence images, and input the dataset and the pose skeleton sequence images into the image dynamic generation model for training to obtain the trained image dynamic generation model. Specifically, DWPose is generally used to extract the pose sequence images of the characters (controlling the body and hands), and OpenPose is used to render them into pose skeleton sequence images.

[0052] The static image to be generated is obtained and input into the trained image dynamic generation model. The trained image dynamic generation model obtains the character reference image features according to the static image, and generates a dynamic video according to the character reference image features and the posture skeleton sequence image. The posture sequence is initially encoded using the posture guide and fused with multi-frame noise, and then denoised by the denoising network to generate a video.

[0053] Specifically: The integration of reference images involves two aspects: first, extracting detail features through ReferenceNet and using them for spatial attention, and second, extracting semantic features through the CLIP image encoder for cross attention. The computational block of the denoising network consists of spatial attention, cross attention, and temporal attention, such as Figure 1As shown in the dashed box on the right. Compared with the reference image integration, the denoising network adds temporal attention to the original two aspects, and the temporal attention operates in the time dimension. After denoising, the VAE decoder decodes the result into a video clip.

[0054] Further, the character reference image features are obtained according to the static image, including:

[0055] Extracting detail features of static images for spatial attention;

[0056] Extract semantic features of static images through image encoder for cross attention;

[0057] The detail features and semantic features are fused to generate the character reference graph features.

[0058] In the text-to-video task, the textual cues explicitly express high-level semantics and only need to be semantically relevant to the generated visual content. However, in the image-to-video task, the images encapsulate more low-level detail features, requiring precise consistency of the generated results. In previous studies focusing on image-driven generation, most methods use the CLIP image encoder instead of the text encoder in cross-attention. However, this design does not address issues related to detail consistency. One reason for this limitation is that the input of the CLIP image encoder contains low-resolution (224×224) images, resulting in the loss of important fine-grained detail information. Another factor is that CLIP is trained to match the semantic features of text, emphasizing high-level feature matching, which leads to the loss of detailed features in feature encoding.

[0059] Therefore, a reference image feature extraction network named ReferenceNet is designed. The same framework as the denoising network is adopted for ReferenceNet, excluding the temporal layer. Similar to the denoising network, ReferenceNet inherits weights from the original SD and updates the weights of each SD independently. The method of integrating the features in ReferenceNet into the denoising network is as follows: Figure 1 As shown, the self-attention layer is replaced by a spatial attention layer.

[0060] Given the feature maps x1∈R t×h×w×c from the denoising network and x2∈R h×w×c from the ReferenceNet, x2 is first replicated t times and concatenated with x1 along the w dimension. Self-attention is then performed to extract the first half of the feature map as the output. This design has two advantages: first, the ReferenceNet can leverage the pre-trained image feature modeling capabilities of the original SD, resulting in a well-initialized feature. Second, since the ReferenceNet and the denoising network have essentially the same network structure and shared initialization weights, the denoising network can selectively learn relevant features in the same feature space from the ReferenceNet. In addition, cross-attention is performed using the CLIP image encoder. By leveraging the feature space shared with the text encoder, it provides the semantic features of the reference image as a beneficial initialization to speed up the training process of the entire network.

[0061] Although ReferenceNet introduces a comparable number of parameters as the denoising network, in diffusion-based video generation, all video frames are denoised multiple times, while ReferenceNet only needs to extract features once during the entire process. Therefore, it does not lead to a significant increase in computational overhead during inference.

[0062] Furthermore, generating dynamic video based on the character reference image features and posture skeleton sequence images includes:

[0063] Extract detailed features of the character reference image features and use them for spatial attention;

[0064] Extract semantic features of character reference graph features for cross attention;

[0065] Determine the temporal relationship of actions based on the sequence of posture skeleton images for temporal attention;

[0066] The output of combining spatial attention, cross attention, and temporal attention is decoded into dynamic video clips.

[0067] For the temporal attention layer, many studies have proposed incorporating a complementary temporal layer into the text-to-image (T2I) model to capture the temporal dependencies between video frames. This design promotes the transfer of pre-trained image generation capabilities in the T2I base model. Following this principle, the temporal layer is integrated after the spatial attention and cross-attention components in the Res-Trans block. Specifically, for a feature map x∈Rb×t×h×w×c, it is first reshaped into x∈R(b×h×w)×t×c, and then temporal attention is performed, where temporal attention refers to self-attention along dimension t. Features from the temporal layer are merged into the original features through residual connections. The temporal layer is only applied within the Res-Trans block of the denoising network. For ReferenceNet, it computes features of a single reference image and does not participate in temporal modeling. Since Pose guide achieves the controllability of continuous motion of characters, experiments show that the temporal layer guarantees the temporal smoothness and continuity of appearance details, thereby avoiding complex motion modeling.

[0068] In another embodiment, a pose sequence of a character in a data set is extracted, and the pose sequence is rendered to obtain a pose skeleton sequence image, including: using a convolutional layer to align the pose sequence image of a video clip in the data set with the noise at the same resolution, and adding the processed pose sequence image to the noise hidden layer to obtain the pose skeleton sequence image. Specifically, Pose Guider uses four convolutional layers (4×4 kernel, 2×2 strides, using 16, 32, 64, 128 channels) to align the pose image with the same resolution as the noise potential. Subsequently, the processed pose image is added to the noise hidden layer and then input into the denoising network. Pose Guider initializes Gaussian weights and uses zero convolution in the final projection layer.

[0069] The quality evaluation results of the generated results using PSNR, SSIM and FVD on the UBC fashion video public dataset are shown in Table 1. The method proposed in the present invention shows significant advantages, especially in video metrics (FVD).

[0070] Table 1 Video quality evaluation results

[0071]

[0072] On the other hand, the present invention provides a dynamic video generation system based on a diffusion model, comprising:

[0073] The acquisition module is used to obtain several character pictures and video clips to build a data set;

[0074] Construction module, constructs an image dynamic generation model based on the diffusion model;

[0075] The sequence generation module extracts the pose sequence images of the characters in the data set and renders the pose sequence to obtain the pose skeleton sequence images;

[0076] A training module, which inputs the data set and the posture skeleton sequence images into the image dynamic generation model for training, and obtains a trained image dynamic generation model;

[0077] The generation module obtains the static image to be generated and inputs it into the trained image dynamic generation model. The trained image dynamic generation model obtains the character reference image features according to the static image, and generates a dynamic video according to the character reference image features and the posture skeleton sequence image.

[0078] Preferably, the generation module includes:

[0079] A reference image feature extraction unit, used for obtaining a character reference image feature according to a static image;

[0080] A denoising unit is used to generate dynamic videos based on the character reference image features and pose skeleton sequence images.

[0081] Preferably, the reference image feature extraction unit includes:

[0082] Feature extraction layer, used to extract detailed features of static images for spatial attention;

[0083] Image encoder, used to extract semantic features of static images for cross-attention;

[0084] The fusion unit is used to fuse the detail features and the semantic features to generate the character reference graph features.

[0085] Preferably, the denoising unit comprises:

[0086] The spatial attention layer is used to extract the detailed features of the character reference image features and use them for spatial attention;

[0087] Cross-attention layer, used to extract semantic features of the character reference graph features for cross-attention;

[0088] The temporal attention layer is used to determine the temporal relationship of actions based on the pose skeleton sequence images for temporal attention;

[0089] The video decoder decodes the output of the combined spatial attention layer, cross attention layer, and temporal attention layer into dynamic video clips.

[0090] Preferably, the sequence generation module includes:

[0091] Convolutional layer, used to align the pose sequence images of the video clips in the dataset with the noise at the same resolution;

[0092] The noise hidden layer is used to obtain a posture skeleton sequence image according to the processed posture sequence image.

[0093] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0094] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating dynamic video based on a diffusion model, characterized in that: include: Get several character pictures and video clips to form a data set; Construct an image dynamic generation model based on the diffusion model; Extracting posture sequence images of the characters in the data set, rendering the posture sequence images to obtain posture skeleton sequence images, and inputting the data set and the posture skeleton sequence images into the image dynamic generation model for training to obtain a trained image dynamic generation model; The static image to be generated is obtained and input into the trained image dynamic generation model. The trained image dynamic generation model obtains the character reference image features according to the static image, and generates a dynamic video according to the character reference image features and posture skeleton sequence images.

2. The method for generating dynamic video based on a diffusion model according to claim 1, characterized in that: Acquiring the character reference image features according to the static image includes: Extracting detail features of the static image for spatial attention; Extracting semantic features of the static image through an image encoder for cross attention; The detail features and the semantic features are fused to generate character reference graph features.

3. The method for generating dynamic video based on a diffusion model according to claim 1, characterized in that: Generating a dynamic video according to the character reference image features and the posture skeleton sequence images includes: Extracting detailed features of the character reference image features and using them for spatial attention; Extracting semantic features of the character reference graph features for cross attention; Determining the temporal relationship of actions for temporal attention according to the posture skeleton sequence images; The output of combining spatial attention, cross attention, and temporal attention is decoded into dynamic video clips.

4. The method for generating dynamic video based on a diffusion model according to claim 1, characterized in that: Extracting the pose sequence of the character in the data set, rendering the pose sequence to obtain the pose skeleton sequence image, including: using a convolutional layer to align the pose sequence image of the video clip in the data set with the noise at the same resolution, and adding the processed pose sequence image to the noise hidden layer to obtain the pose skeleton sequence image.

5. A dynamic video generation system based on a diffusion model, characterized in that: include: The acquisition module is used to obtain several character pictures and video clips to form a data set; Construction module, constructs an image dynamic generation model based on the diffusion model; A sequence generation module extracts posture sequence images of the characters in the data set, and renders the posture sequence images to obtain posture skeleton sequence images; A training module, inputting the data set and the posture skeleton sequence image into the image dynamic generation model for training to obtain a trained image dynamic generation model; The generation module obtains the static image to be generated and inputs it into the trained image dynamic generation model. The trained image dynamic generation model obtains the character reference image features according to the static image, and generates a dynamic video according to the character reference image features and the posture skeleton sequence image.

6. A dynamic video generation system based on a diffusion model according to claim 5, characterized in that: The generation module comprises: A reference image feature extraction unit, used to obtain a character reference image feature according to the static image; A denoising unit is used to generate a dynamic video based on the character reference image features and posture skeleton sequence images.

7. The dynamic video generation system based on diffusion model according to claim 6, characterized in that: The reference image feature extraction unit comprises: A feature extraction layer, used to extract detail features of the static image for spatial attention; An image encoder, for extracting semantic features of the static image for cross attention; A fusion unit is used to fuse the detail feature and the semantic feature to generate a character reference graph feature.

8. The dynamic video generation system based on diffusion model according to claim 6, characterized in that: The denoising unit comprises: A spatial attention layer, used to extract detailed features of the character reference image features and use them for spatial attention; A cross attention layer, used for extracting semantic features of the character reference graph features for cross attention; A temporal attention layer, for determining a temporal relationship of actions for temporal attention according to the sequence of posture skeleton images; The video decoder decodes the output results of the spatial attention layer, the cross attention layer, and the temporal attention layer into a dynamic video clip.

9. The dynamic video generation system based on diffusion model according to claim 5, characterized in that: The sequence generation module comprises: A convolutional layer, used to align the posture sequence images of the video clips in the data set with the noise at the same resolution; The noise hidden layer is used to obtain a posture skeleton sequence image according to the processed posture sequence image.

Images