Multi-reference fusion driven human body video generation method and related equipment

Abstract

The invention discloses a multi-reference fusion driving human body video generation method and related equipment, a dual-network framework is constructed, and the dual-network framework comprises five main components: a posture guider, a reference network, a denoising network, a multi-reference fusion module and a reference score network. Wherein the attitude guider is responsible for extracting attitude features from an input attitude sequence; the reference network is responsible for extracting reference features from a group of input reference images; the multi-reference fusion module is responsible for fusing the reference features; the reference score network is responsible for scoring the fused reference features and sampling the fused reference features according to scores; the de-noising network is responsible for executing a de-noising task and generating a final video sequence in combination with information from the attitude guider and the reference network. According to the method, the reference features extracted from the plurality of reference images are fused and sampled, so that an efficient and simple reference feature form is obtained, and a human body action video with higher fidelity and better appearance consistency is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and in particular to a multi-reference fusion driven human body video generation method and related equipment. Background Art

[0002] Controllable human action video generation is a challenging task in computer vision with many potential applications, such as short entertainment videos, film art creation, game production, digital humans, virtual characters, and data generation.

[0003] To achieve controllable generated results, existing methods typically input a reference human image and an action sequence into a generative network, which then drives the reference image and generates a human action video sequence. In this field, two main approaches have dominated: those based on generative adversarial networks (GANs) and diffusion models.

[0004] Traditional GAN-based methods typically rely on a warping module to spatially transform a reference image to align it with the target pose, thereby generating animated videos. While GAN's image generation capabilities can be leveraged to inpaint invisible regions of the reference image, ensuring the credibility of these regions remains a significant challenge, especially when the target pose differs significantly from the reference pose. Furthermore, GAN-based methods face limitations in generalization performance, making it difficult to generalize to data beyond the training data, which significantly limits their application.

[0005] Recently, generative diffusion models have demonstrated strong capabilities and generalization performance in generating high-quality images and videos. Trained on large-scale image data, diffusion image models have achieved impressive results in terms of visual quality and diversity. This provides a strong visual diffusion prior for various visual generation tasks, such as image inpainting, image editing, and video generation. This strong visual diffusion prior makes it possible to generate high-quality human motion videos. Researchers have begun exploring the use of diffusion model architectures to drive motion video generation from human images. Due to the strong generalization capabilities of generative diffusion models, researchers have recently favored using pre-trained diffusion models to generate motion videos from a single human image. Diffusion-based methods can be divided into two categories based on appearance extraction techniques. The first category typically employs a pre-trained coarse-grained global encoder (e.g., CLIP) to extract coarse-grained appearance features and aligns these appearance features into the latent space via a cross-attention mechanism. The second category typically replicates an initialized denoising network as a reference network. The reference network extracts fine-grained reference features from a reference image and feeds these features into the denoising process via a cross-model attention mechanism. These two approaches typically load a pre-trained image generation network (such as Stable Diffusion) as initialization parameters. Because pre-trained models lack the temporal modeling capabilities required for video generation tasks, these approaches typically insert a temporal module into the network, expanding the image generation model into a video generation model. Compared to GANs, diffusion-based approaches significantly improve the authenticity and continuity of generated results, as well as their generalizability beyond the training data.

[0006] However, existing diffusion model-based methods only consider driving a single reference human image, which makes the generated results usually limited by the invisible areas of the reference image. Figure 1 As shown in the figure, when it is necessary to generate areas that are not present in the reference image or are occluded (such as palms, clothing logos, etc.), existing methods will produce low-quality results such as artifacts, which undermines the authenticity and usability of the generated video and fails to faithfully restore the appearance of the human body in the reference image. This approach of relying on a single reference image makes it difficult to maintain a consistent reference human appearance when generating new human poses that are different from the reference image. In practical applications, in order to generate good and acceptable results, existing methods rely on carefully selecting high-quality reference images, which greatly limits the scope of application of this technology. For example, it is not always possible for users to provide high-quality reference images (full-body photos with complete details of all body parts). Even if users can provide high-quality full-body photos as reference images, existing methods find it difficult to generate results with significantly changed human poses (such as pictures with the back facing the camera) based on such reference images.

[0007] To address this issue, a simple approach is to introduce multiple reference images. However, in existing approaches, directly introducing multiple reference images into the input significantly increases the scale of both the input sequence and the reference feature sequence, significantly increasing computational overhead. Overly large reference features also make it difficult for the denoising network to efficiently utilize them, reducing the performance benefits of introducing multiple reference images. Summary of the Invention

[0008] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a multi-reference fusion driven human body video generation method and related equipment.

[0009] The first technical solution adopted by the present invention is:

[0010] A multi-reference fusion-driven human body video generation method comprises the following steps:

[0011] S1, obtain the human skeleton posture sequence p and a set of reference images r;

[0012] S2. Construct a variational autoencoder and initialize it with pre-trained parameters; the autoencoder includes an encoder E and a decoder D;

[0013] S3. Build denoising network N D , reference network N R , attitude guide N P , multi-reference fusion module N F and the reference score network N S , and load the preset parameters θ;

[0014] S4. Randomly sample an input noise z from the standard normal distribution N(0,1) T ;

[0015] S5. Input the human skeleton posture sequence p into the posture guide N P In the figure, we obtain the posture feature f p ;

[0016] S6. Input the reference image r into the reference network N R In the above example, we obtain multi-level reference features f r ;

[0017] S7, multi-level reference features f r Input to the multi-reference fusion module N F In the fusion reference feature f rf ;

[0018] S8, fuse the reference feature f rf Input to the reference score network N SScore the reference importance score s;

[0019] S9, convert the reference importance score s into a probability distribution and fusion reference feature f rf Sampling is performed to obtain the sampling reference feature f rc ;

[0020] S10, input noise z T Input to the denoising network N D The input layer obtains the intermediate noise feature f m ;

[0021] S11, the intermediate noise feature f m and posture feature f p Add together to obtain the posture noise feature f mp , in order to achieve the effect of posture-guided generation;

[0022] S12, the posture noise feature f mp The input is fed into the stacked denoising Transformer block and compared with the sampled reference feature f rc Fusion is performed to obtain the output feature f o ;

[0023] S13, output feature f o Input to the denoising network N D The output layer of , we get the prediction noise ε;

[0024] S14, according to the predicted noise ε input noise z T Perform denoising to obtain denoising feature z T-1 ;

[0025] S15, using denoising feature z T-1 Replace the original z T As the denoising network N D Repeat steps S10-S15 until denoising is completed and the final denoising feature z0 is obtained;

[0026] S16. Input the denoising feature z0 into the decoder D of the variational autoencoder to obtain a video frame sequence v in the image space.

[0027] Furthermore, the preset parameter θ is obtained by:

[0028] A1. Extract one or more video data from a preset data set as current training data;

[0029] A2. Use a human posture detector to detect the human skeleton posture video from the original video;

[0030] A3. Sample a human video sequence h and a set of reference images r from the original video, and sample a human skeleton pose sequence p from the human skeleton pose video;

[0031] A4. Input the human video sequence f into the encoder E of the variational autoencoder to obtain its potential feature z0;

[0032] A5. Randomly sample the diffusion time step t and random noise ε0, add noise to the latent feature z0, and obtain the noise latent feature z t ;

[0033] A6. Input the human skeleton posture sequence p into the posture guide N P In the figure, we obtain the posture feature f p ;

[0034] A7. Input the reference image r into the reference network N R In the above example, we obtain multi-level reference features f r ;

[0035] A8, multi-level reference features f r Input to the multi-reference fusion module N F In the fusion reference feature f rf ;

[0036] A9. Fusion reference feature f rf Input into the reference score network for scoring to obtain the reference importance score s;

[0037] A10. According to the reference importance score s, the fusion reference feature f rf Perform Gumbel-softmax sampling to obtain the sampling reference feature f rc ;

[0038] A11, the noise potential feature z t Input to the denoising network N D The input layer N Dinput , obtain the intermediate noise feature f m ;

[0039] A12, the intermediate noise feature f m and posture feature f p Add together to obtain the posture noise feature f mp , in order to achieve the effect of posture-guided generation;

[0040] A13, the posture noise feature f mp The input is fed into the stacked denoising Transformer block and compared with the sampled reference feature f rc Fusion is performed to obtain the output feature f o ;

[0041] A14, output feature f o Input to the denoising network N D The output layer N Doutput , get the prediction noise ε;

[0042] A15. Calculate the loss function between the predicted noise ε and the random noise ε0, and jointly optimize the denoising network N D , reference network N R , attitude guide N P , multi-reference fusion module N F and the reference score network N S ;

[0043] A16. Repeat steps A1-A16 until the training is completed, and save the denoising network N after training. D , reference network N R , attitude guide N P , multi-reference fusion module N F and the reference score network N s The parameter θ.

[0044] Furthermore, the denoising network N D and reference network N R is constructed as follows:

[0045] Transformer is used as the network structure. Each Transformer network consists of several cascaded Transformer blocks. The denoising network N D and reference network N R There are the same number L of Transformer blocks, each of which contains a multi-head self-attention module and a feed-forward network.

[0046] Furthermore, the diffusion time step t is injected into the Transformer block using an adaptive layer normalization method, which learns a set of scaling parameters α, γ and displacement parameters β for normalization. The scaling parameters γ and displacement parameters β are used before each attention module and feedforward network, and the scaling parameter α is used before the residual connection.

[0047] In the denoising network N D A temporal attention mechanism is added after the spatial attention of each Transformer block in order to improve the temporal consistency and continuity of the generated results.

[0048] Furthermore, the posture guide N PIt is a neural network containing multiple convolutional layers. All convolutional layers except the last one are randomly initialized, and the last convolutional layer is zero-initialized, that is, all its parameters are initialized to 0.

[0049] Furthermore, the multi-reference fusion module N F It includes a set of reference fusion units N Fi , that is, N F =[N F1 ,…,N FL ]; Each multi-reference fusion unit N Fi is a self-attention module, L is the number of reference fusion modules;

[0050] Given a multi-reference feature Where R represents the number of reference frames, B represents the batch size, and C represents the number of channels; first reshape it into Then the attention mechanism is performed on the spatial dimension L of the reshaped reference feature, as follows:

[0051]

[0052] Q=Q1⊕Q2⊕…⊕Q R

[0053] K=K1⊕K2⊕…⊕K R

[0054] V=V1⊕V2⊕…⊕V R

[0055] in Represents z r The i-th feature in , ⊕ represents vector connection.

[0056] Furthermore, the reference score network N S Includes L reference score units N Si , each reference score unit N Si It is a network with two linear layers and two activation layers, namely N S =[N S1 ,…,N SL ].

[0057] The second technical solution adopted by the present invention is:

[0058] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a multi-reference fusion-driven human body video generation method as described above.

[0059] The third technical solution adopted by the present invention is:

[0060] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, which is loaded and executed by a processor to implement a multi-reference fusion-driven human body video generation method as described above.

[0061] The fourth technical solution adopted by the present invention is:

[0062] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above-mentioned multi-reference fusion-driven human body video generation method.

[0063] The present invention achieves the following beneficial effects: by fusing and sampling reference features extracted from multiple reference images, an efficient and streamlined reference feature representation is obtained. The denoising network can further utilize the multiple reference features extracted in this way, thereby generating human action videos with higher fidelity and more consistent appearance. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0065] Figure 1 It is a comparison chart of the advantages and disadvantages of single reference and multiple reference;

[0066] Figure 2 This is a diagram of the overall network architecture of a multi-reference fusion-driven human body video generation method according to an embodiment of the present invention;

[0067] Figure 3 This is a diagram of the Transformer block structure in an embodiment of the present invention;

[0068] Figure 4 This is a comparison chart of the denoising time between the present invention and the direct injection method;

[0069] Figure 5This is a comparison chart of the qualitative results of the present invention and other methods on the TikTok dataset. DETAILED DESCRIPTION

[0070] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application. For the step numbers in the following embodiments, they are provided only for the convenience of explanation and are not intended to limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0071] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise clearly defined, words such as setting, installing, and connecting should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0072] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0073] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0074] In the description of this application, "and / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0075] To address existing technical issues, the present invention proposes a multi-reference fusion-driven technology, which comprises two main mechanisms: a multi-reference feature fusion mechanism and a multi-reference feature sampling mechanism. By fusing and sampling reference features extracted from multiple reference images, an efficient and streamlined reference feature form is obtained. The denoising network can make full use of the multi-reference features extracted by this method, thereby generating human motion videos with higher fidelity and better appearance consistency. Compared with existing methods, the present invention's method achieves significant improvements in technical indicators such as FID, FID-VID, and FVD.

[0076] Example 1

[0077] like Figure 2 As shown, the present invention implements a dual-network framework comprising five main components: a pose guide, a reference network, a denoising network, a multi-reference fusion module, and a reference score network. The pose guide extracts pose features from an input pose sequence; the reference network extracts reference features from a set of input reference images; the multi-reference fusion module fuses the reference features; the reference score network scores the fused reference features and samples them based on the scores; and the denoising network performs the denoising task, combining information from the pose guide and reference network to generate the final video sequence.

[0078] As an implementation method, the network model is trained through the following steps:

[0079] Step 1: Construct a variational autoencoder, including encoder E and decoder D, and initialize them with pre-trained parameters.

[0080] The role of the variational autoencoder is to map the input image sequence to features in the latent space through the encoder E, and then reconstruct the features in the latent space into an image through the decoder D.

[0081] Step 2: Build the denoising network N D and reference network N R , and initialized with pre-trained parameters.

[0082] For example, the denoising network N D and reference network N R The specific network structure used is not limited and can be CNN, UNet, Transformer, etc. In our implementation, Transformer is used as the network structure. Each Transformer network consists of several cascaded Transformer blocks. The denoising network N D and reference network N R Transformer blocks with the same number L. Figure 3 As shown, each Transformer block contains a multi-head self-attention module and a feed-forward network.

[0083] The operation process of multi-head self-attention is:

[0084] 1) Input: sequence Where n is the sequence length, d model is the model dimension;

[0085] 2) Linear transformation: Generate query (Q), key (K), value (V) matrices:

[0086] Q=xW Q ,K=xW K ,V=xW V

[0087] in

[0088] 3) Split features: Split the Q, K, and V matrices into h heads along the feature dimension, and the dimension of each head is d k =d model / h;

[0089] 4) Parallel computing: Each head performs self-attention calculation independently to generate The output:

[0090]

[0091] 5) Merge and linear transformation: concatenate the outputs of all heads and pass them through the linear layer Fusion information.

[0092] The feedforward network is located after the self-attention layer and is usually composed of two linear transformation layers and an activation function. Its mathematical form is:

[0093] FFN(x)=ReLU(xW1+b1)W2+b2

[0094] The input features Weight Matrix b1 and b2 are bias terms, d ffn The middle dimension.

[0095] This example uses a method called adaptive layer normalization to inject the diffusion time step t into the Transformer block. This method learns a set of scaling parameters α, γ, and displacement parameters β for normalization. We use the scaling parameters γ and displacement parameters β before each attention module and feedforward network, and the scaling parameter α before the residual connection. The formula is as follows:

[0096]

[0097] In order to improve the temporal consistency and continuity of the generated results, we D The temporal attention mechanism is added after the spatial attention of each Transformer block in . Given a spatial feature We reformulate it into And perform self-attention along the time dimension T, incorporating temporal features into spatial features through residual connections. This design promotes information exchange between frames, which is crucial for realizing the network's temporal modeling capabilities.

[0098] Step 3: Build the Pose Guide N P , and initialize the parameters.

[0099] For example, the attitude director N P The specific network structure used is not limited, and can be CNN, UNet, Transformer, etc. In our implementation, the posture guide N P It is a neural network containing multiple convolutional layers. We randomly initialize all convolutional layers except the last one, and zero-initialize the last convolutional layer, that is, initialize all its parameters to 0.

[0100] Step 4: Build a set of multi-reference fusion modules N F and the reference score network N S , and initialize the parameters.

[0101] Multi-reference fusion module N F and the reference score network N S The specific network structure used is not limited, and can be MLP, CNN, UNet, Transformer, etc. In our implementation, each multi-reference fusion module N Fi It is a self-attention module, based on the reference network N constructed in the previous article R Given the number of Transformer blocks L in the dataset, we construct a set of multi-reference fusion modules N with the same number of F =[N F1 ,…,N FL ] and randomly initialize its parameters. Given a multi-reference feature Where R represents the number of reference frames, we first reshape it into Then the attention mechanism is performed on the spatial dimension L of the reshaped reference feature, as follows:

[0102]

[0103] Q=Q1⊕Q2⊕…⊕Q R

[0104] K=K1⊕K2⊕…⊕K R

[0105] V=V1⊕V2⊕…⊕V R

[0106] in Represents Z r The i-th feature in (B×R)×L×C is represented by ⊕, which represents vector concatenation. This approach effectively extracts and fuses multi-view information from multiple reference features with significantly different spatial layouts. The (B×R)×L×C representation has an additional bracket compared to B×R×L×C, indicating that the first two dimensions will be merged, resulting in four dimensions before merging and three dimensions after merging.

[0107] In our implementation, each reference score network N S It is a network with two linear layers and two activation layers. According to the reference network N constructed in the previous article R Given the number of Transformer blocks L in the dataset, we construct a set of reference score networks N with the same number of S =[N S1 ,…,N SL ] and randomly initialize its parameters.

[0108] Step 5: Extract one or more video data from the dataset as current training data.

[0109] Theoretically, any human action video can be used as training data, but the quality of the training data will affect the network's performance. Constructing a high-quality, clean dataset of single-person action videos can significantly improve the network's ultimate performance. In this example, we used the TikTok dataset from the open source community and a self-collected video dataset. The TikTok dataset contains approximately 350 videos, while the self-collected dataset contains approximately 4,000 videos, each approximately 10-15 seconds long. For the TikTok dataset, we center-cropped the images and resized the resolution to 512×512; for the self-collected dataset, we center-cropped the images and resized the resolution to 512×288. We uniformly normalized all images, with the mean and variance being [0.5, 0.5, 0.5] and [0.5, 0.5, 0.5], respectively.

[0110] Step 6: Use a human pose detector to detect the human skeleton pose video from the original video.

[0111] Specifically, any existing human posture detector can be used for detection. In this embodiment, a human skeleton sequence is extracted from a video using an existing human skeleton detector DWPose.

[0112] Step 7: Sample a human video sequence h and a set of reference images r from the original video, and sample a human skeleton pose sequence p from the human skeleton pose video.

[0113] In one embodiment, based on a set video sequence length, we randomly sample a continuous segment of frames of corresponding length from the original video as the human body video sequence h, and use the same sampling index to sample the human body skeleton pose sequence p from the body skeleton pose video. We divide the video into R segments and randomly sample one frame from each segment as the reference frame, where R is the number of reference frames. In our experiments, we set the number of reference frames R = 4 and the video sequence length to 16.

[0114] Step 8: Input the human video sequence f into the encoder E of the variational autoencoder to obtain its potential feature z0:

[0115] z0=E(f)

[0116] in In our implementation, c ′ =4, k=8.

[0117] Step 9: Randomly sample diffusion time step t and random noise ε0, add noise to the latent feature z0, and obtain the noise latent feature z t .

[0118] Specifically, we randomly sample an integer t from [1, T], in our implementation, T = 1000. We randomly sample a noise ε0 from the standard normal distribution N(0, 1) and add the noise to the latent feature z0 to obtain the noisy latent feature z t .

[0119] Step 10: Input the human skeleton pose sequence p into the pose guide N P In the figure, we obtain the posture feature f p :

[0120] f p =N P (p)

[0121] Step 11: The reference image r is input to the reference network N R In the above example, we obtain multi-level reference features f r .

[0122] Specifically, the reference image passes through the reference network N in sequence. RIn the L cascaded Transformer blocks, we take the feature f in each Transformer block before entering the self-attention module ri Save it, and then get the multi-level reference feature f r =[f r1 ,…,f rL ].

[0123] Step 12: Multi-level reference features f r Input to the multi-reference fusion module N F In the fusion reference feature f rf :

[0124] f rfi =N Fi (f ri )

[0125] where i = 1,…,L, f rf =[f rf1 ,…,f rfL ].

[0126] Step 13: Fusion reference feature f rf Input into the reference score network for scoring and obtain the reference importance score s:

[0127] s i =N Si (f rfi )

[0128] where i=1,…,L,s i =[s1,…,s L ].

[0129] Step 14: Fusion reference feature f according to reference importance score s rf Perform Gumbel-softmax sampling to obtain the sampling reference feature f rc .

[0130] Since the sampling operation is non-differentiable, in order to meet the full-link differentiability requirements of network training, we use Gumbel-Softmax to convert the importance scores into one-hot weights w = [w1,…,w L ], and then perform weighted summation on multiple reference features:

[0131]

[0132] Step 15: Substitute the noise latent feature z t Input to the denoising network N D The input layer N Dinput , obtain the intermediate noise feature f m :

[0133] f m =N Dinput (z t )

[0134] Step 16: The intermediate noise feature f m and posture feature f p Add together to obtain the posture noise feature f mp , to achieve the effect of posture-guided generation:

[0135] f mp =f m +f p

[0136] Step 17: Transform the pose noise feature f mp The input is fed into the stacked denoising Transformer block and compared with the sampled reference feature f rc Fusion is performed to obtain the output feature f o .

[0137] Cross-network spatial attention is to transfer reference information from the reference network N R Injected into the denoising network N D The key components of , which work as follows: For the reference feature We repeat it T times and reshape it into Where L represents the spatial dimension and T represents the time dimension. We then compare the reshaped reference features with the noise features. Connect along the spatial dimension L and perform spatial self-attention, taking the first half of the output features as the final output features. This method can effectively inject the reference features into the noise features, making the denoising network N D Ability to generate results that match the reference image.

[0138] Step 18: Output feature f o Input to the output layer N of the denoising network Doutput , and get the prediction noise ε:

[0139] ε=N Dou t put (f o )

[0140] Step 19: Calculate the loss function between the predicted noise ε and the random noise ε0, and jointly optimize the denoising network N D , reference network N R , attitude guide N P , multi-reference fusion module N F and the reference score network N S .

[0141] Specifically, various forms of loss functions can be used for optimization. In this embodiment, the loss function is:

[0142]

[0143] Step 20: Repeat steps 5 to 20 until the training is completed, and save the denoising network N after training. D , reference network N R , attitude guide N P , multi-reference fusion module N F and the reference score network N S The parameter θ.

[0144] After model training is completed, the trained model can be used to generate videos. The specific inference process is as follows:

[0145] Step 1: The user provides a human skeleton pose sequence p and a set of reference images r.

[0146] Specifically, the reference images r provided by the user must be different images of the same human individual, and the user can use an existing human pose detector to extract the required human skeleton pose sequence p from other human action videos.

[0147] Step 2: Construct a variational autoencoder, including encoder E and decoder D, and initialize them with pre-trained parameters.

[0148] Step 3: Build the denoising network N D , reference network N R , attitude guide N P , multi-reference fusion module N F and the reference score network N S , and load the parameters θ saved after the previous training.

[0149] Step 4: Randomly sample an input noise z from the standard normal distribution N(0,1) T .

[0150] Step 5: Input the human skeleton pose sequence p into the pose guide N P In the figure, we obtain the posture feature f p :

[0151] f p =N P (p)

[0152] Step 6: Input the reference image r into the reference network N R In the above example, we obtain multi-level reference features f r .

[0153] Specifically, the reference image passes through the reference network N in sequence.R In the L cascaded Transformer blocks, we take the feature f in each Transformer block before entering the self-attention module ri Save it, and then get the multi-level reference feature f r =[f r1 ,…,f rL ].

[0154] Step 7: Multi-level reference features f r Input to the multi-reference fusion module N F In the fusion reference feature f rf :

[0155] f rf i =N F i (f r i )

[0156] where i = 1,…,L, f rf =[f rf1 ,…,f rfL ].

[0157] Step 8: Fusion reference feature f rf Input into the reference score network for scoring and obtain the reference importance score s:

[0158] s i =N S i (f rf i )

[0159] where i=1,…,L,s i =[s1,…,s L ].

[0160] Step 9: Convert the reference importance score s into a probability distribution and fused the reference feature f rf Sampling is performed to obtain the sampling reference feature f rc :

[0161] f rc =sample(f rf , s)

[0162] Step 10: Convert the input noise z T Input to the denoising network N D The input layer obtains the intermediate noise feature f m :

[0163] f m =N Dinput(z t )

[0164] Step 11: Transform the intermediate noise feature f m and posture feature f p Add together to obtain the posture noise feature f mp , to achieve the effect of posture-guided generation:

[0165] f mp =f m +f p

[0166] Step 12: Transform the posture noise feature f mp The input is fed into the stacked denoising Transformer block and compared with the sampled reference feature f rc Fusion is performed to obtain the output feature f o .

[0167] Step 13: Output feature f o Input to the output layer of the denoising network to obtain the predicted noise ε:

[0168] ε=N Doutput (f o )

[0169] Step 14: Compare the input noise z to the predicted noise ε T Perform denoising to obtain denoising feature z T-1 :

[0170] z T-1 =Denoising(z T ,ε)

[0171] Denoising is the denoising operation.

[0172] Step 15: Use denoising feature z T=1 Replace the original z T As the denoising network N D Repeat steps 10 to 15 until denoising is completed and the final denoising feature z0 is obtained.

[0173] Step 16: Input the denoising feature z0 into the decoder D of the variational autoencoder to obtain the video frame sequence v in the image space:

[0174] v=D(z0)

[0175] Experimental verification:

[0176] As shown in Table 1, the method of the present invention is quantitatively compared with other methods using TikTok data. The indicators used include FID, FID-VID, and FVD. The calculation method of these three indicators is to first extract image or video features, calculate the mean and covariance matrix of the feature vector on the features, and finally calculate the distance between the feature distributions of the generated data and the real data. The lower the value, the smaller the distance between the distributions, and the generated data is more consistent with the real data distribution.

[0177] Table 1 Comparison data of the present invention and other methods on the TikTok dataset

[0178] FID FID-VID FVD DreamPose 72.62 78.77 551.02 Disco 30.75 59.90 292.08 Disco+ 28.31 55.17 267.75 Animate Anyone - - 171.90 Magic Pose 25.50 46.30 - The present invention 20.27 16.16 97.38

[0179] Assuming d represents the channel dimension of the feature, n represents the sequence length of a single image embedding, r represents the number of reference frames, and T represents the number of denoising steps, the computational time complexity of directly injecting multiple reference features is The computational complexity increases rapidly with the growth of r. After the introduction of the multi-reference feature sampling mechanism, the computational time complexity is There is no additional overhead compared to using a single reference. Figure 4 This figure shows a comparison of the computational cost of the present invention compared to the direct injection method. It can be seen that as the number of reference images increases, the direct injection method experiences a significant increase in denoising time, while the present invention method maintains almost the same denoising time, demonstrating the present invention's computational cost advantage.

[0180] Figure 5 The qualitative comparison between the proposed method and other methods on the TikTok dataset is shown. Our method has greater advantages in detail preservation and authenticity. Even small logos on clothes can be faithfully restored by our method, which is impossible for previous methods.

[0181] Example 2

[0182] An embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the following Figure 2 A multi-reference fusion driven human video generation method is shown.

[0183] It is understood that the memory may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the server, etc.

[0184] The processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the server. It executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, as well as accessing data stored in memory. Optionally, the processor may be implemented using at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU) and a modem. The CPU primarily processes the operating system and application programs, while the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a single chip.

[0185] Since the electronic device is an electronic device corresponding to a multi-reference fusion driven human body video generation method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0186] Example 3

[0187] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 2 A multi-reference fusion driven human video generation method is shown.

[0188] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0189] Since the storage medium is a storage medium corresponding to a multi-reference fusion-driven human body video generation method in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0190] Example 4

[0191] In some possible implementations, various aspects of the method of the embodiments of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps of the multi-reference fusion-driven human body video generation method according to various exemplary embodiments of the present application described above in this specification. The executable computer program code or "code" for performing each embodiment may be written in a high-level programming language such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0192] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0193] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0194] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made based on the essence of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A multi-reference fusion driven human body video generation method, characterized in that: The following steps are involved: S1, obtain the human skeleton posture sequence p and a set of reference images r; S2. Build a variational autoencoder and initialize it with pre-trained parameters. S3. Build denoising network N D , reference network N R , attitude guide N P , multi-reference fusion module N F and the reference score network N S , and load the preset parameters θ; S4. Randomly sample an input noise z from the standard normal distribution N(0,1) T ; S5. Input the human skeleton posture sequence p into the posture guide N P In the figure, we obtain the posture feature f p ; S6. Input the reference image r into the reference network N R In the above example, we obtain multi-level reference features f r ; S7, multi-level reference features f r Input to the multi-reference fusion module N F In the fusion reference feature f rf ; S8, fuse the reference feature f rf Input to the reference score network N S Score the reference importance score s; S9, convert the reference importance score s into a probability distribution and fusion reference feature f rf Sampling is performed to obtain the sampling reference feature f rc ; S10, input noise z T Input to the denoising network N D The input layer obtains the intermediate noise feature f m ; S11, the intermediate noise feature f m and posture feature f p Add together to obtain the posture noise feature f mp ; S12, the posture noise feature f mp The input is fed into the stacked denoising Transformer block and compared with the sampled reference feature f rc Fusion is performed to obtain the output feature f o ; S13, output feature f o Input to the denoising network N D The output layer of , we get the prediction noise ε; S14, according to the predicted noise ε input noise z T Perform denoising to obtain denoising feature z T-1 ; S15, using denoising feature z T-1 Replace the original z T As the denoising network N D Repeat steps S10-S15 until denoising is completed and the final denoising feature z0 is obtained; S16. Input the denoising feature z0 into the decoder D of the variational autoencoder to obtain a video frame sequence v in the image space.

2. The multi-reference fusion driven human body video generation method according to claim 1, characterized in that: The preset parameter θ is obtained in the following way: A1. Extract one or more video data from a preset data set as current training data; A2. Detect human skeleton posture video from the original video; A3. Sample a human video sequence h and a set of reference images r from the original video, and sample a human skeleton pose sequence p from the human skeleton pose video; A4. Input the human video sequence f into the encoder E of the variational autoencoder to obtain its potential feature z0; A5. Randomly sample the diffusion time step t and random noise ε0, add noise to the latent feature z0, and obtain the noise latent feature z t ; A6. Input the human skeleton posture sequence p into the posture guide N P In the figure, we obtain the posture feature f p ; A7. Input the reference image r into the reference network N R In the above example, we obtain multi-level reference features f r ; A8, multi-level reference features f r Input to the multi-reference fusion module N F In the fusion reference feature f rf ; A9. Fusion reference feature f rf Input into the reference score network for scoring to obtain the reference importance score s; A10. According to the reference importance score s, the fusion reference feature f rf Perform Gumbel-softmax sampling to obtain the sampling reference feature f rc ; A11, the noise potential feature z t Input to the denoising network N D The input layer N Dinput , obtain the intermediate noise feature f m ; A12, the intermediate noise feature f m and posture feature f p Add together to obtain the posture noise feature f mp ; A13, the posture noise feature f mp The input is fed into the stacked denoising Transformer block and compared with the sampled reference feature f rc Fusion is performed to obtain the output feature f o ; A14, output feature f o Input to the denoising network N D The output layer N Doutput , get the prediction noise ε; A15. Calculate the loss function between the predicted noise ε and the random noise ε0, and jointly optimize the denoising network N D , reference network N R , attitude guide N P , multi-reference fusion module N F and the reference score network N S ; A16. Repeat steps A1-A16 until the training is completed, and save the denoising network N after training. D , reference network N R , attitude guide N P , multi-reference fusion module N F and the reference score network N S The parameter θ.

3. A multi-reference fusion driven human body video generation method according to claim 1 or 2, characterized in that: The denoising network N D and reference network N R is constructed as follows: Transformer is used as the network structure. Each Transformer network consists of several cascaded Transformer blocks. The denoising network N D and reference network N R There are the same number L of Transformer blocks, each of which contains a multi-head self-attention module and a feed-forward network.

4. The multi-reference fusion driven human body video generation method according to claim 3, characterized in that: The diffusion time step t is injected into the Transformer block using the adaptive layer normalization method, which learns a set of scaling parameters α, γ and displacement parameters β for normalization. The scaling parameters γ and displacement parameters β are used before each attention module and feedforward network, and the scaling parameter α is used before the residual connection; In the denoising network N D A temporal attention mechanism is added after the spatial attention of each Transformer block in order to improve the temporal consistency and continuity of the generated results.

5. A multi-reference fusion driven human body video generation method according to claim 1 or 2, characterized in that: The attitude guide N P It is a neural network with multiple convolutional layers. All convolutional layers except the last one are randomly initialized, and the last convolutional layer is zero-initialized.

6. A multi-reference fusion driven human body video generation method according to claim 1 or 2, characterized in that: The multi-reference fusion module N F It includes a set of reference fusion units N Fi , that is, N F =[N F1 ,…,N FL ]; Each multi-reference fusion unit N Fi is a self-attention module, L is the number of reference fusion modules; Given a multi-reference feature Where R represents the number of reference frames, B represents the batch size, and C represents the number of channels; first reshape it into Then the attention mechanism is performed on the spatial dimension L of the reshaped reference feature, as follows: in Represents z r The i-th feature in Represents vector connection.

7. A multi-reference fusion driven human body video generation method according to claim 1 or 2, characterized in that: The reference score network N S Includes L reference score units N Si , each reference score unit N Si It is a network with two linear layers and two activation layers, namely N S =[N S1 ,…,N SL ].

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises computer instructions, which are used to perform the method according to any one of claims 1 to 7 when executed by a processor.

Images