Video generation and training methods, devices, equipment and media for video generation models

Abstract

This disclosure provides a method, apparatus, device, and medium for video generation and training a video generation model, relating to the technical fields of computer vision, deep learning, and large models, and applicable to scenarios such as AIGC, digital humans, and intelligent e-commerce. The specific implementation scheme is as follows: acquiring a sequence of action poses and displaying a reference image with a target object; wherein, the elements in the action pose sequence represent action poses generated based on human key points; encoding the reference image to obtain the appearance features of the target object; extracting pose guidance features from the action pose sequence, and generating target features based on the pose guidance features, noise features, and appearance features; decoding the target features to obtain an action-driven video displaying the target object.

Description

Technical Field

[0001] This disclosure relates to the field of AI (Artificial Intelligence), specifically to the technical fields of computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC (Artificial Intelligence Generated Content), digital humans, and intelligent e-commerce, especially to methods, devices, equipment, and media for video generation and training video generation models. Background Technology

[0002] Digital human motion-driven technology applies the motion and posture of a driving object to a digital human (such as a target object), enabling the digital human in the generated motion-driven video to have the same body movements, gestures, facial expressions, and lip movements as the driving object. This high-precision motion and posture transfer not only enhances the realism and interactivity of digital humans but also greatly expands their application scenarios. For example, digital human motion-driven technology can be applied to live streaming, social networking, customer service, and gaming, driving the rapid development of related industries. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and medium for video generation and training video generation models.

[0004] According to one aspect of this disclosure, a video generation method is provided, comprising:

[0005] Acquire a sequence of motion postures and display a reference image of a target object; wherein, the elements in the sequence of motion postures represent motion postures generated based on human key points;

[0006] The reference image is encoded to obtain the appearance features of the target object;

[0007] The posture guidance features are extracted from the action posture sequence, and the target features are generated based on the posture guidance features, noise features, and appearance features.

[0008] The target features are decoded to obtain an action-driven video displaying the target object.

[0009] According to another aspect of this disclosure, a method for training a video generation model is provided, comprising:

[0010] Acquire an action video and the corresponding action posture sequence, as well as acquire a reference image displaying the target object; wherein, the elements in the action posture sequence represent action postures generated based on human body key points in the video frames of the action video;

[0011] The action video is encoded to obtain a first video feature, and noise is added to the first video feature to obtain a second video feature;

[0012] The action posture sequence is encoded to obtain posture guidance features, and the reference image is encoded to obtain the appearance features of the target object;

[0013] A video generation model is used to generate predictive noise based on the second video features, the pose guidance features, and the appearance features;

[0014] The video generation model is trained based on the difference between the added noise to the first video feature and the predicted noise.

[0015] According to another aspect of this disclosure, a video generation apparatus is provided, comprising:

[0016] The acquisition module is used to acquire a sequence of action postures and display a reference image of a target object; wherein, the elements in the sequence of action postures represent action postures generated based on human key points;

[0017] An encoding module is used to encode the reference image to obtain the appearance features of the target object;

[0018] The extraction module is used to extract posture guidance features from the action posture sequence;

[0019] The generation module is used to generate target features based on the posture guidance features, noise features, and appearance features;

[0020] The decoding module is used to decode the target features to obtain an action-driven video displaying the target object.

[0021] According to another aspect of this disclosure, a training apparatus for a video generation model is provided, comprising:

[0022] The acquisition module is used to acquire an action video and the corresponding action posture sequence, as well as to acquire a reference image displaying the target object; wherein, the elements in the action posture sequence represent the action postures generated based on human body key points in the video frames of the action video.

[0023] An encoding and noise-adding module is used to encode the action video to obtain a first video feature, and to add noise to the first video feature to obtain a second video feature;

[0024] The feature extraction module is used to encode the action posture sequence to obtain posture guidance features, and to encode the reference image to obtain the appearance features of the target object;

[0025] The generation module is used to generate predictive noise using a video generation model based on the second video features, the pose guidance features, and the appearance features;

[0026] The training module is used to train the video generation model based on the difference between the added noise of the first video features and the predicted noise.

[0027] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0028] At least one processor; and

[0029] A memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores instructions that can be executed by the at least one processor, which, when executed, enable the at least one processor to perform either the video generation method proposed in one aspect of this disclosure or the training method for the video generation model proposed in another aspect of this disclosure.

[0031] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, the computer instructions being used to cause the computer to perform the video generation method proposed in one aspect of this disclosure, or to perform the training method of the video generation model proposed in another aspect of this disclosure.

[0032] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the video generation method proposed in one aspect of this disclosure, or, when executed, implements the training method for the video generation model proposed in another aspect of this disclosure.

[0033] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0034] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0035] Figure 1 This is a flowchart illustrating the video generation method provided in Embodiment 1 of this disclosure;

[0036] Figure 2 This is a flowchart illustrating the video generation method provided in Embodiment 2 of this disclosure;

[0037] Figure 3This is a schematic diagram illustrating the model structure and video generation principle of the video generation model provided in the embodiments of this disclosure;

[0038] Figure 4 This is a flowchart illustrating the video generation method provided in Embodiment 3 of this disclosure;

[0039] Figure 5 This is a flowchart illustrating the video generation method provided in Embodiment 4 of this disclosure;

[0040] Figure 6 This is a flowchart illustrating the video generation method provided in Embodiment 5 of this disclosure;

[0041] Figure 7 This is a flowchart illustrating the training method for the video generation model provided in Embodiment Six of this disclosure;

[0042] Figure 8 This is a schematic diagram of the video generation apparatus provided in Embodiment 7 of this disclosure;

[0043] Figure 9 This is a schematic diagram of the structure of the training device for the video generation model provided in Embodiment 8 of this disclosure;

[0044] Figure 10 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0045] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0046] With the rapid development of AIGC technology, digital humans, as a new type of human-computer interaction, are attracting increasing attention. Currently, digital human motion-driven technology mainly focuses on lip movements, while body movements are derived from a recorded substrate. This results in a mismatch between the digital human's voice, tone, and physical actions when driving the digital human, compromising its realism.

[0047] Digital humans can be divided into 2D (two-dimensional) digital humans and 3D (three-dimensional) digital humans. 2D digital humans have many advantages such as low cost and simple production.

[0048] In related technologies, high-quality 2D digital humans are mainly achieved through deep learning technology. When driving digital humans, the focus is mainly on lip movement. For motion driving of digital humans, it can be achieved based on intelligent agents. By inputting 2D key points of the human body, stable video diffusion is used as a pre-trained model to generate and display motion-driven videos of digital humans.

[0049] However, the above models are mainly designed for general intelligent agent scenarios, and the generation quality and realism are generally average. Hand artifacts are clearly visible, and the video timing is unstable, making them unusable in digital human scenarios.

[0050] Therefore, in response to at least one of the aforementioned problems, this disclosure proposes a method, apparatus, device, and medium for video generation and training a video generation model.

[0051] The following description, with reference to the accompanying drawings, outlines a method, apparatus, device, and medium for video generation and training a video generation model according to embodiments of this disclosure. Before detailing the embodiments of this disclosure, commonly used technical terms will be introduced for ease of understanding:

[0052] Large models are machine learning models with a massive number of parameters and complex computational structures, typically built from deep neural networks and possessing billions or even hundreds of billions of parameters. The purpose of large models is to improve their expressive power and predictive performance, enabling them to handle more complex tasks and data. Large models have wide applications in various fields, including natural language processing, computer vision, speech recognition, and recommender systems.

[0053] Figure 1 This is a flowchart illustrating the video generation method provided in Embodiment 1 of this disclosure.

[0054] This disclosure illustrates the example of the video generation method being configured in a video generation apparatus, which can be applied to any electronic device to enable the electronic device to perform video generation functions.

[0055] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0056] like Figure 1 As shown, the video generation method may include the following steps S101 to S104:

[0057] Step S101: Obtain the action posture sequence and display the reference image with the target object; wherein, the elements in the action posture sequence represent the action posture generated based on human body key points.

[0058] The action pose sequence, also known as a keypoint sequence or human keypoint sequence, consists of elements that represent action poses generated based on human keypoints. For example, human keypoints can be 2D keypoints, which are easier to obtain and more robust.

[0059] The target object can be any user object.

[0060] In the embodiments of this disclosure, there are no restrictions on the method of obtaining the action posture sequence and reference image. For example, the action posture sequence and reference image can be provided manually by relevant personnel, or the action posture sequence and reference image can be collected online, such as by using web crawler technology to collect the action posture sequence and reference image online, or the action posture sequence and reference image can be synthesized manually, etc. The embodiments of this disclosure do not limit this.

[0061] Step S102: Encode the reference image to obtain the appearance features of the target object.

[0062] Among them, appearance features are used to indicate the texture details of the target object.

[0063] In this embodiment of the disclosure, the appearance of the target object in the reference image can be encoded based on feature encoding technology to obtain the appearance features of the target object.

[0064] As an example, a reference image can be input into the encoder in a pre-trained VAE (Variational AutoEncoder) to encode the reference image and obtain the appearance features of the latent space.

[0065] A Generative Imagery (VAE) is a generative model that combines the ideas of an autoencoder and variational Bayesian inference. It learns a latent representation of input data and can generate new, similar data samples based on this latent representation. A VAE consists of an encoder and a decoder. The encoder maps the input data to points in a latent space, i.e., the latent representation. The decoder generates new, similar data samples based on this latent representation.

[0066] Step S103: Extract posture guidance features from the action posture sequence, and generate target features based on posture guidance features, noise features, and appearance features.

[0067] There are no restrictions on the method of acquiring noise features. For example, noise features can be collected online, or noise features can be artificially synthesized, or noise features can be obtained by randomly sampling a set distribution (such as Gaussian distribution, uniform distribution, Poisson distribution, mixed distribution, etc.), and so on.

[0068] It should be noted that, due to the good mathematical properties, ease of implementation, and smoothness of the Gaussian distribution, random sampling of the Gaussian distribution to obtain noise features can make the noise features independent and identically distributed. This helps the model learn the continuity and smooth changes of the data, improves the quality of video generation, and helps the model better capture the statistical characteristics of the data, thus avoiding overfitting.

[0069] In this embodiment of the disclosure, posture guidance features can be extracted from the action posture sequence based on feature extraction technology.

[0070] As an example, an encoder in a VAE can be used to encode the action pose sequence to obtain pose guidance features. Alternatively, a pose encoder can be used to encode the action pose sequence to obtain pose guidance features. Or, a pose guider network can be used to extract features from the action pose sequence to obtain pose guidance features.

[0071] In this embodiment of the disclosure, target features can be generated by integrating posture guidance features, noise features, and appearance features. For example, deep learning techniques can be used to fuse posture guidance features, noise features, and appearance features to obtain the target features.

[0072] Step S104: Decode the target features to obtain an action-driven video showing the target object.

[0073] In this embodiment of the disclosure, target features can be decoded based on feature decoding technology to obtain an action-driven video displaying the target object.

[0074] As an example, a decoder in a VAE can be used to decode the target features to obtain motion-driven video. This motion-driven video displays a target object whose pose matches the pose indicated by a motion pose sequence. Therefore, it is possible to generate a video of the target object's actions based on a given motion pose sequence.

[0075] The video generation method of this disclosure automatically generates a motion-driven video of the target object by inputting a reference image displaying the target object, a sequence of motion postures (or a sequence of human key points) and noise features. The motion posture of the target object in the motion-driven video is consistent with the motion posture indicated by the motion posture sequence, which can improve the realism of the target object in the motion-driven video, that is, improve the video generation quality and improve the user experience in the digital human-driven scenario.

[0076] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0077] To clearly illustrate how target features are generated based on posture guidance features, noise features, and appearance features in any embodiment of this disclosure, this disclosure also proposes a video generation method.

[0078] Figure 2 This is a flowchart illustrating the video generation method provided in Embodiment 2 of this disclosure.

[0079] like Figure 2 As shown, the video generation method may include the following steps S201 to S206:

[0080] Step S201: Obtain the action posture sequence and display the reference image with the target object; wherein, the elements in the action posture sequence represent the action posture generated based on human key points.

[0081] Step S202: Encode the reference image to obtain the appearance features of the target object.

[0082] Step S203: Extract posture guidance features from the action posture sequence.

[0083] The explanation of steps S201 to S203 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0084] Step S204: Input the pose guidance features, noise features, and appearance features into the video generation model to perform feature processing using multiple diffusion modules in the video generation model.

[0085] As one possible approach, if the length of the action pose sequence is not greater than a set length threshold, the pose guidance features, noise features, and appearance features can be directly input into the video generation model simultaneously, so that multiple cascaded diffusion modules in the video generation model can be used for feature processing.

[0086] The set length threshold can be a pre-set length threshold. For example, this set length threshold can be set according to the input requirements of the video generation model.

[0087] In any embodiment of this disclosure, each diffusion module may include a pose modulation unit, a spatial self-attention unit, and a temporal self-attention unit, wherein the pose modulation unit is used to modulate the pose of the input features; the spatial attention unit is used to denoise the input features based on a spatial attention mechanism (or a spatial self-attention mechanism); and the temporal attention unit is used to denoise the input features based on a temporal attention mechanism (or a temporal self-attention mechanism).

[0088] As an example, if the length of the action pose sequence does not exceed a set length threshold, the first diffusion module in a series of concatenated diffusion modules is used to perform steps A through D:

[0089] Step A: The attitude modulation unit modulates the noise features based on the attitude guidance features to obtain the modulation features of the first diffusion module, so that the modulation features can be effectively injected into the key point information of the human body.

[0090] The noise features have the same dimensions as the posture guidance features.

[0091] Step B: Combine the modulation features and appearance features of the first diffusion module to obtain the combined features of the first diffusion module.

[0092] As an example, the modulation features and appearance features of the first diffusion module can be concatenated along the token (a token is the smallest unit of data processing, which can refer to an image block, pixel group, or feature vector) dimension to obtain the concatenated features of the first diffusion module.

[0093] Step C: Use spatial attention units to denoise the spliced ​​features of the first diffusion module to obtain the intermediate features of the first diffusion module.

[0094] The spatial attention unit (or spatial attention layer) is used to learn how to transfer a reference image to a specified sequence of action poses.

[0095] For example, the spatial attention unit can denoise the spliced ​​features of the first diffusion module based on the spatial attention mechanism to obtain the intermediate features of the first diffusion module.

[0096] Step D: Use a temporal attention unit to denoise the intermediate features of the first diffusion module to obtain the output features of the first diffusion module.

[0097] In this disclosure, in order to ensure the temporal stability of video generation, a spatial self-attention mechanism can be used to improve the continuity of the video.

[0098] For example, the temporal attention unit can denoise the intermediate features of the first diffusion module based on the temporal attention mechanism to obtain the output features of the first diffusion module.

[0099] As an example, a non-first diffusion module in a series of diffusion modules (e.g., the i-th diffusion module, where i is a positive integer greater than 1) is used to perform steps A' to D':

[0100] Step A': The attitude modulation unit modulates the output features of the (i-1)th diffusion module based on the attitude guidance features to obtain the modulation features of the ith diffusion module.

[0101] Step B': The modulation feature and appearance feature of the i-th diffusion module are concatenated to obtain the concatenated feature of the i-th diffusion module. The implementation principle is similar to that of Step B, and will not be elaborated here.

[0102] Step C': Use spatial attention units to denoise the spliced ​​features of the i-th diffusion module to obtain the intermediate features of the i-th diffusion module. The implementation principle is similar to step C, and will not be elaborated here.

[0103] Step D': Denoise the intermediate features of the i-th diffusion module using a temporal attention unit to obtain the output features of the i-th diffusion module. The implementation principle is similar to step D and will not be elaborated here.

[0104] In summary, feature processing using pose modulation, spatial attention mechanisms, and temporal attention mechanisms can improve the temporal stability, continuity, and generation quality of video, thereby enhancing the user experience in digital human-driven scenarios.

[0105] In any embodiment of this disclosure, the spatial attention unit in each diffusion module can be used to denoise the input features using a deterministic denoising strategy and a spatial attention mechanism.

[0106] In any embodiment of this disclosure, the temporal attention unit in each diffusion module can be used to denoise the input features using a deterministic denoising strategy and a temporal attention mechanism.

[0107] In any embodiment of this disclosure, the spatial attention unit in each diffusion module can be used to denoise the input features using a deterministic denoising strategy and a spatial attention mechanism, and the temporal attention unit in each diffusion module can also be used to denoise the input features using a deterministic denoising strategy and a temporal attention mechanism.

[0108] Therefore, adopting a deterministic denoising strategy to denoise features step by step can reduce the disturbance in the denoising process and further improve the quality of video generation.

[0109] Step S205: Determine the target features based on the output features of the last diffusion module.

[0110] As one possible implementation, if the length of the action posture sequence is not greater than a set length threshold, the output feature of the last diffusion module in a series of diffusion modules can be used as the target feature.

[0111] Step S206: Decode the target features to obtain an action-driven video showing the target object.

[0112] The explanation of step S206 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0113] As an example, the model structure and principles of video generation can be described as follows: Figure 3 As shown, the video generation model comprises N cascaded DiT (Diffusion Models with Transformers) modules. Each DiT module can include a pose modulation unit, a spatial self-attention unit, and a temporal self-attention unit. Figure 3 In this context, "Reshape" refers to changing the shape of the feature tensor to adapt to the input requirements of the next network layer.

[0114] The video generation principle is as follows: The reference image is input into the encoder in the VAE for encoding to obtain appearance features. The action-driven sequence is input into the Pose Guider network for feature extraction to obtain pose guidance features. The pose modulation unit in the first DiT module modulates the noise features based on the pose guidance features to obtain modulation features. Then, the modulation features are concatenated with the appearance features and sequentially input into the cascaded spatial self-attention unit and temporal self-attention unit for progressive denoising to obtain the output features of the first DiT module.

[0115] The output features of the first DiT module are used to replace the noise features and are input into the second DiT module. The pose modulation unit in the second DiT module modulates the output features of the first DiT module based on the pose guidance features to obtain the modulated features. Then, the modulated features are concatenated with the appearance features and sequentially input into the cascaded spatial self-attention unit and temporal self-attention unit for progressive denoising to obtain the output features of the second DiT module.

[0116] By analogy, the output features of the Nth DiT module can be obtained, and the decoder in the input VAE can be used to decode the output features of the Nth DiT module to obtain the motion-driven video.

[0117] The video generation method of this disclosure employs multiple diffusion modules to progressively denoise the input features. Each diffusion module can focus on removing a portion of the noise, thereby refining the generation results layer by layer and gradually improving the quality of the generated video. In other words, each denoising step can be regarded as an optimization of the previous step's result. This progressive optimization process helps to generate more refined and realistic videos, enhancing the stability and generalization ability of the model.

[0118] To clearly illustrate the above embodiments, this disclosure also proposes a video generation method.

[0119] Figure 4 This is a schematic flowchart of the video generation method provided in Embodiment 3 of this disclosure.

[0120] like Figure 4 As shown, the video generation method may include the following steps S401 to S406:

[0121] Step S401: Obtain the action posture sequence and display the reference image with the target object; wherein, the elements in the action posture sequence represent the action posture generated based on human body key points.

[0122] Step S402: Encode the reference image to obtain the appearance features of the target object.

[0123] The explanation of steps S401 to S402 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0124] Step S403: In response to the length of the action posture sequence being greater than a set length threshold, the action posture sequence is split into multiple action posture sub-sequences; wherein, adjacent action posture sub-sequences contain overlapping elements.

[0125] The set length threshold can be a pre-set length threshold. For example, this set length threshold can be set according to the input requirements of the video generation model.

[0126] It should be noted that setting this length threshold based on the input requirements of the video generation model has at least the following advantages: reduced memory consumption (lower memory requirements for each forward and backward propagation); improved computational efficiency (supporting parallel and batch processing, accelerating training and inference); avoidance of gradient vanishing and exploding, improving model stability through local gradient calculation; simplified model design (converting long sequences into fixed-length subsequences simplifies model design and implementation); improved model generalization ability, helping the model learn local features; and flexible attention mechanisms (more effectively applying local and global attention mechanisms).

[0127] In this embodiment of the disclosure, when the length of the action posture sequence is greater than a set length threshold, the action sequence can be split based on the set length threshold to obtain multiple action posture sub-sequences. The length of each action posture sub-sequence is not greater than the set length threshold, and adjacent action posture sub-sequences contain overlapping elements.

[0128] Step S404: Extract the pose guidance features of multiple action pose sub-sequences respectively.

[0129] In this embodiment of the disclosure, the posture guidance sub-features of each action posture sub-sequence can be extracted based on feature extraction technology.

[0130] As an example, an encoder in a VAE can be used to encode each action pose subsequence to obtain pose guidance sub-features. Alternatively, a pose encoder can be used to encode each action pose subsequence to obtain pose guidance sub-features. Alternatively, a pose guider network can be used to extract features from each action pose subsequence to obtain pose guidance sub-features.

[0131] Step S405: Generate target features based on multiple pose guidance sub-features, noise features, and appearance features.

[0132] In this embodiment of the disclosure, deep learning technology can be used to generate target features by integrating pose guidance sub-features, noise features, and appearance features.

[0133] Step S406: Decode the target features to obtain an action-driven video showing the target object.

[0134] The explanation of step S406 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0135] The video generation method of this disclosure, when dealing with long action pose sequences, segments the sequence into sub-sequences. These sub-sequences can be processed in parallel, improving video generation efficiency. Furthermore, the segmented sub-sequences help the model better learn local features, which can be combined to form global features, enhancing the model's generalization ability and robustness. In addition, overlapping elements between adjacent sub-sequences ensure smooth transitions between generated actions, avoiding abrupt jumps or breaks, further improving video generation quality and producing more refined and realistic videos.

[0136] To clearly illustrate how target features are generated based on multiple pose guidance sub-features, noise features, and appearance features in the above embodiments of this disclosure, this disclosure also proposes a video generation method.

[0137] Figure 5 This is a flowchart illustrating the video generation method provided in Embodiment 4 of this disclosure.

[0138] like Figure 5 As shown, the video generation method may include the following steps S501 to S508:

[0139] Step S501: Obtain the action posture sequence and display the reference image with the target object; wherein, the elements in the action posture sequence represent the action posture generated based on human key points.

[0140] Step S502: Encode the reference image to obtain the appearance features of the target object.

[0141] Step S503: In response to the length of the action posture sequence being greater than a set length threshold, the action posture sequence is split into multiple action posture subsequences.

[0142] There are overlapping elements between adjacent action pose subsequences.

[0143] Step S504: Extract the pose guidance features of multiple action pose sub-sequences respectively.

[0144] The explanation of steps S501 to S504 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0145] Step S505: Input any pose guidance sub-feature, noise feature and appearance feature into the video generation model to perform feature processing using multiple diffusion modules in the video generation model.

[0146] In any embodiment of this disclosure, each diffusion module may include a pose modulation unit, a spatial self-attention unit, and a temporal self-attention unit, wherein the pose modulation unit is used to modulate the pose of the input features, the spatial attention unit is used to denoise the input features based on the spatial attention mechanism, and the temporal attention unit is used to denoise the input features based on the temporal attention mechanism.

[0147] As an example, the first diffusion module in a series of cascaded diffusion modules is used to perform steps E through H:

[0148] Step E: For any one of the multiple posture guidance sub-features, the posture modulation unit modulates the noise feature based on the posture guidance sub-feature to obtain the modulation feature of the first diffusion module, so that the modulation feature can effectively inject human key point information.

[0149] The noise features have the same dimension as the attitude guidance sub-features.

[0150] Step F: Combine the modulation features and appearance features of the first diffusion module to obtain the combined features of the first diffusion module.

[0151] Step G: Use a spatial attention unit based on the spatial attention mechanism to denoise the spliced ​​features of the first diffusion module to obtain the intermediate features of the first diffusion module.

[0152] The spatial attention unit (or spatial attention layer) is used to learn how to transfer a reference image to a specified sequence of action poses.

[0153] Step H: Use a temporal attention unit to denoise the intermediate features of the first diffusion module based on the temporal attention mechanism to obtain the output features of the first diffusion module.

[0154] In this disclosure, in order to ensure the temporal stability of video generation, a spatial self-attention mechanism can be used to improve the continuity of the video.

[0155] As an example, a non-first diffusion module in a series of diffusion modules (e.g., the i-th diffusion module, where i is a positive integer greater than 1) is used to perform steps E' to H':

[0156] Step E': The attitude modulation unit modulates the output features of the (i-1)th diffusion module based on the attitude guidance sub-feature to obtain the modulation features of the i-th diffusion module.

[0157] Step F': The modulation feature and appearance feature of the i-th diffusion module are concatenated to obtain the concatenated feature of the i-th diffusion module. The implementation principle is similar to that of step F, and will not be elaborated here.

[0158] Step G': Use a spatial attention unit based on the spatial attention mechanism to denoise the spliced ​​features of the i-th diffusion module to obtain the intermediate features of the i-th diffusion module.

[0159] Step H': Use a temporal attention unit to denoise the intermediate features of the i-th diffusion module based on the temporal attention mechanism to obtain the output features of the i-th diffusion module.

[0160] In summary, feature processing using pose modulation, spatial attention mechanisms, and temporal attention mechanisms can improve the temporal stability, continuity, and generation quality of video, thereby enhancing the user experience in digital human-driven scenarios.

[0161] In any embodiment of this disclosure, the attitude modulation unit in each diffusion module is used to perform attitude modulation on a portion of the attitude guidance sub-features of adjacent action attitude sub-sequences using the same initialization noise; wherein the portion of the features is obtained by encoding overlapping elements.

[0162] Therefore, by using the same initial noise to process the overlapping features, a smooth transition between adjacent subsequences can be ensured, avoiding abrupt changes at the boundaries of the generated results and improving the overall coherence and contextual consistency of the generated video. Furthermore, by maintaining the consistency of the overlapping features, detailed information in the video can be better preserved, resulting in a more refined, realistic, and coherent video.

[0163] In any embodiment of this disclosure, the spatial attention unit and / or temporal attention unit in each diffusion module are further configured to smooth the features corresponding to overlapping elements during the denoising time step.

[0164] In summary, smoothing ensures smoother feature changes between adjacent subsequences, avoiding abrupt changes or discontinuities at boundaries in the generated results, maintaining temporal continuity of features, and making the generated video smoother and more natural. Furthermore, smoothing helps preserve details in overlapping areas, avoiding detail loss due to noise or discontinuities, and better recovering texture and structural information in the video, resulting in more refined and realistic videos. In addition, smoothing enhances the alignment of features in overlapping areas, ensuring consistency of these features across different subsequences, and improves the signal-to-noise ratio of features, allowing the model to better focus on the true features of the data.

[0165] Step S506: Use the output feature of the last diffusion module as the target sub-feature corresponding to any pose guidance sub-feature.

[0166] In this embodiment of the disclosure, the output feature of the last diffusion module in a plurality of serially connected diffusion modules can be used as the target sub-feature corresponding to the above-mentioned attitude guidance sub-feature.

[0167] Step S507: The target sub-features corresponding to multiple pose guidance features are fused to obtain the target features.

[0168] In this embodiment of the disclosure, target features can be obtained by fusing target sub-features corresponding to multiple pose guidance features based on feature fusion technology.

[0169] Step S508: Decode the target features to obtain an action-driven video showing the target object.

[0170] The explanation of step S508 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0171] The video generation method of this disclosure employs multiple diffusion modules to progressively denoise the input features. Each diffusion module can focus on removing a portion of the noise, thereby refining the generation results layer by layer and gradually improving the quality of the generated video. In other words, each denoising step can be regarded as an optimization of the previous step's result. This progressive optimization process helps to generate more refined and realistic videos, enhancing the stability and generalization ability of the model.

[0172] To clearly illustrate how target features are generated based on multiple pose guidance sub-features, noise features, and appearance features in the above embodiments of this disclosure, this disclosure also proposes a video generation method.

[0173] Figure 6 This is a flowchart illustrating the video generation method provided in Embodiment 5 of this disclosure.

[0174] like Figure 6 As shown, the video generation method may include the following steps S601 to S608:

[0175] Step S601: Obtain the action posture sequence and display the reference image with the target object; wherein, the elements in the action posture sequence represent the action posture generated based on human body key points.

[0176] Step S602: Encode the reference image to obtain the appearance features of the target object.

[0177] Step S603: In response to the length of the action posture sequence being greater than a set length threshold, the action posture sequence is split into multiple action posture subsequences.

[0178] There are overlapping elements between adjacent action pose subsequences.

[0179] Step S604: Extract the pose guidance features of multiple action pose sub-sequences respectively.

[0180] The explanation of steps S601 to S604 can be found in the relevant descriptions in any embodiment of this disclosure, and will not be repeated here.

[0181] Step S605: Input any pose guidance sub-feature, noise feature and appearance feature into the video generation model to perform feature processing using multiple diffusion modules in the video generation model.

[0182] The implementation principle of step S605 is similar to that of step S505, and will not be elaborated here.

[0183] Step S606: Use the output features of the last diffusion module as the target features.

[0184] In this embodiment of the disclosure, the output feature of the last diffusion module in a plurality of serially connected diffusion modules can be directly used as the target feature.

[0185] Step S607: Decode multiple target features to obtain multiple video segments.

[0186] In this embodiment of the disclosure, multiple target features can be decoded separately to obtain multiple video segments.

[0187] As an example, a decoder in a VAE can be used to decode each target feature to obtain a video segment.

[0188] Step S608: Combine multiple video clips to obtain an action-driven video displaying a target object.

[0189] In this embodiment of the disclosure, multiple video segments can be synthesized based on video synthesis technology to obtain a motion-driven video displaying a target object. The motion posture of the target object in the motion-driven video is consistent with the motion posture indicated by the motion posture sequence, thereby enabling the generation of a video showing the target object's motion based on a given motion posture sequence.

[0190] The video generation method of this disclosure employs multiple diffusion modules to process multiple pose guidance sub-features respectively, which can better preserve the detailed information in each sub-sequence, making the generated video more refined and realistic. Furthermore, each diffusion module can focus on removing a portion of noise, enabling the generation result to be refined layer by layer, gradually improving the quality of the generated video. That is, each step of denoising can be regarded as an optimization of the previous step's result. This progressive optimization process helps to generate more refined and realistic videos, enhancing the stability and generalization ability of the model.

[0191] By decoding multiple target features separately, the detailed information in each sub-sequence can be better preserved, making the generated video clips more refined and realistic. By synthesizing multiple video clips, the feature changes between adjacent sub-sequences can be made smoother, avoiding abrupt changes at the boundaries of the generated motion-driven video. This makes the synthesized motion-driven video more continuous in time, and the transition of actions more natural and smooth.

[0192] The above are various embodiments of the application method (i.e., video generation method) of the video generation model. This disclosure also proposes a training method for the video generation model.

[0193] Figure 7 This is a flowchart illustrating the training method for the video generation model provided in Embodiment Six of this disclosure.

[0194] like Figure 7 As shown, the training method for this video generation model may include the following steps S701 to S705:

[0195] Step S701: Obtain the motion video and the corresponding motion posture sequence, as well as the reference image displaying the target object; wherein, the elements in the motion posture sequence represent the motion postures generated based on the human body key points in the video frames of the motion video.

[0196] The target object can be any user object.

[0197] In this embodiment of the disclosure, there are no restrictions on the method of obtaining the reference image. For example, the reference image can be obtained from the training set or the test set, or the reference image can be provided manually by the user, or the reference image can be collected online, or the target object can be image collected to obtain the reference image, or the reference image can be synthesized manually, etc. This embodiment of the disclosure does not limit this.

[0198] In this embodiment of the disclosure, there are no restrictions on the method of acquiring the motion video. For example, the motion video can be acquired from the training set or the test set, or the motion video can be captured online, or the motion video can be obtained by shooting the driving object, or the motion video can be synthesized manually, etc. This embodiment of the disclosure does not limit these methods.

[0199] In this embodiment, the driving object and the target object in the motion video may be the same object or they may be different objects, and this disclosure does not impose any restrictions on this.

[0200] For example, when the driving object and the target object are the same object, the reference image may be a video frame in the motion video.

[0201] In this embodiment of the disclosure, key point detection can be performed on the driving object in each frame of the action video to obtain human key point information of the driving object in each frame of the video, and an action posture sequence can be generated based on the human key point information of the driving object in each frame of the video; wherein, the elements in the action posture sequence are used to represent the action posture generated based on the human key points in the video frame.

[0202] Step S702: Encode the action video to obtain the first video feature, and add noise to the first video feature to obtain the second video feature.

[0203] In this embodiment of the disclosure, action videos can be encoded using feature encoding techniques to obtain video latent space features, referred to as the first video features in this disclosure. For example, an encoder in a pre-trained VAE can be used to encode the action video to obtain the first video features.

[0204] In this embodiment of the disclosure, noise can also be added to the first video feature to obtain the second video feature.

[0205] Step S703: Encode the action posture sequence to obtain posture guidance features, and encode the reference image to obtain the appearance features of the target object.

[0206] Among them, appearance features are used to indicate the texture details of the target object.

[0207] In this embodiment of the disclosure, posture guidance features can be extracted from the action posture sequence based on feature extraction technology.

[0208] As an example, an encoder in a VAE can be used to encode the action pose sequence to obtain pose guidance features. Alternatively, a pose encoder can be used to encode the action pose sequence to obtain pose guidance features. Or, a pose guider network can be used to extract features from the action pose sequence to obtain pose guidance features.

[0209] In this embodiment of the disclosure, the appearance of the target object in the reference image can also be encoded based on feature encoding technology to obtain the appearance features of the target object.

[0210] As an example, a reference image can be input into the encoder in a pre-trained VAE, and the encoder in the VAE can be used to encode the reference image to obtain the appearance features of the latent space.

[0211] Step S704: Using a video generation model, predictive noise is generated based on the second video features, pose guidance features, and appearance features.

[0212] In this embodiment of the disclosure, the second video features, pose guidance features, appearance features, and noise step (used to indicate the degree of noise addition) can be input together into the video generation model for noise prediction to obtain the predicted noise output by the video generation model.

[0213] Step S705: Train the video generation model based on the difference between the added noise and the predicted noise of the first video features.

[0214] In this embodiment of the disclosure, the value of the loss function (referred to as the loss value in this disclosure) can be determined based on the difference between the added noise and the predicted noise of the first video feature. Thus, in this disclosure, the model parameters in the video generation model can be adjusted based on the loss value to minimize the loss value.

[0215] The loss value is positively correlated with the above differences; that is, the greater the difference, the greater the loss value, and vice versa.

[0216] It should be noted that the above example only uses minimizing the loss value as the termination condition for model training. In actual applications, other termination conditions can be set, such as: the training time reaches a set time, the number of training rounds reaches a set number of rounds, etc.

[0217] As one possible implementation method, Figure 7 The video generation model trained in the illustrated embodiment can be used to implement... Figures 1 to 6The video generation method described in any of the embodiments.

[0218] The training method for the video generation model in this embodiment can improve the prediction accuracy of the video generation model by performing supervised training, thereby improving the quality of the video output by the video generation model and enhancing the user experience in digital human-driven scenarios.

[0219] In any embodiment of this disclosure, a Transformer-based diffusion model is used as an example for video generation. A reference image and a sequence of motion poses (i.e., a sequence of 2D human key points) can be input into the diffusion model to generate a high-quality motion-driven video, thereby achieving the matching of motion and speech and improving the naturalness of the digital human.

[0220] During the model training phase: Given an action video and its corresponding pose sequence (i.e., a 2D human keypoint sequence), the action video is input into the encoder in a pre-trained VAE for encoding, yielding the first video feature. Noise is then added to the first video feature to obtain the second video feature. Simultaneously, the pose sequence is input into the encoder in the VAE for encoding, yielding pose guidance features (or keypoint features) with the same dimension as the first video feature. Furthermore, the encoder in the VAE extracts texture details from a given reference image displaying the target object, obtaining the appearance feature of the target object. Then, the second video feature, pose guidance feature, appearance feature, and the noise-adding step are input together into a diffusion model. The diffusion model predicts the added noise, and is trained based on the difference between the predicted noise and the added noise in the first video feature.

[0221] The diffusion model is based on the Transformer architecture and contains N DiT modules. Each DiT module mainly consists of three parts: a pose modulation unit, a spatial self-attention unit, and a temporal self-attention unit. Noise is predicted using these N identical DiT modules, and a loss value is calculated based on the predicted and actual noise levels. This loss value is then used to update the diffusion model.

[0222] The pose modulation unit modulates noise features using pose guidance features to obtain modulated features. Since the pose guidance features and noise features have the same dimension, they can effectively inject key human body information. Then, the modulated features and appearance features are concatenated along the token dimension and input into the spatial self-attention unit. At this point, different video frames of the action video are in the batch dimension. This layer learns how to transfer reference images to a specified action pose sequence. To ensure the temporal stability of video generation, features at the same location from different video frames can be input into the temporal self-attention unit, improving the continuity of the generated video through the spatial self-attention mechanism.

[0223] In the model inference or testing phase: given the sequence of motion poses to be generated and a reference image showing the target object, noise is randomly sampled in a Gaussian distribution to obtain noise features. Denoising is then performed through iterative multi-step steps to output the final motion-driven video. Since each output video segment has a limited length, several strategies can be employed to improve video stability during denoising to enhance the motion continuity of long videos. First, the long motion pose sequence is divided into overlapping subsequences. During video generation, the same initialization noise is used for frames with the same motion to improve consistency. Simultaneously, a deterministic denoising process (such as DDIM (Denoising Diffusion Implicit Models)) is employed to reduce perturbations in the denoising process. In the denoising time step, overlapping regions are smoothed to further improve video continuity. Finally, the output target features are input into the decoder in the VAE for decoding, ultimately resulting in a long-term stable digital human-driven video, referred to in this disclosure as the motion-driven video.

[0224] In summary, by utilizing pose modulation, spatial attention mechanisms, and temporal attention mechanisms, the generation effect of videos can be improved. Furthermore, by employing a series of smoothing and stabilizing denoising strategies, the continuity of the generated videos can be enhanced, enabling the generation of high-quality digital human motion-driven videos and improving the user experience in video generation scenarios.

[0225] With the above Figures 1 to 6 Corresponding to the video generation method provided in the embodiments, this disclosure also provides a video generation apparatus. Since the video generation apparatus provided in the embodiments of this disclosure is similar to the one described above... Figures 1 to 6 The video generation method provided in the embodiments corresponds to the video generation method provided in the embodiments of this disclosure, and therefore the implementation of the video generation method is also applicable to the video generation apparatus provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0226] Figure 8 This is a schematic diagram of the video generation apparatus provided in Embodiment 7 of this disclosure.

[0227] like Figure 8 As shown, the video generation device 800 may include: an acquisition module 810, an encoding module 820, an extraction module 830, a generation module 840, and a decoding module 850.

[0228] The acquisition module 810 is used to acquire a sequence of motion postures and display a reference image of a target object; wherein the elements in the sequence of motion postures represent motion postures generated based on human key points.

[0229] The encoding module 820 is used to encode the reference image to obtain the appearance features of the target object;

[0230] Extraction module 830 is used to extract posture guidance features from action posture sequences;

[0231] The generation module 840 is used to generate target features based on posture guidance features, noise features, and appearance features;

[0232] The decoding module 850 is used to decode the target features to obtain a motion-driven video displaying the target object.

[0233] In one possible implementation of this disclosure, the generation module 840 is configured to: input pose guidance features, noise features, and appearance features into a video generation model to perform feature processing using multiple diffusion modules in the video generation model; and determine target features based on the output features of the last diffusion module.

[0234] In one possible implementation of this disclosure, each diffusion module includes an attitude modulation unit, a spatial attention unit, and a temporal attention unit. In response to the length of the action posture sequence not exceeding a set length threshold, the first diffusion module among the multiple diffusion modules is configured to: use the attitude modulation unit to perform attitude modulation on the noise features based on the attitude guidance features to obtain the modulation features of the first diffusion module; concatenate the modulation features of the first diffusion module with the appearance features to obtain the concatenated features of the first diffusion module; use the spatial attention unit to denoise the concatenated features of the first diffusion module to obtain the intermediate features of the first diffusion module; and use the temporal attention unit to denoise the intermediate features of the first diffusion module to obtain the output features of the first diffusion module.

[0235] In one possible implementation of this disclosure, a non-first diffusion module among multiple diffusion modules is used to: use an attitude modulation unit to perform attitude modulation on the output features of the previous diffusion module based on attitude guidance features to obtain the modulation features of the non-first diffusion module; concatenate the modulation features of the non-first diffusion module with appearance features to obtain the concatenated features of the non-first diffusion module; use a spatial attention unit to denoise the concatenated features of the non-first diffusion module to obtain the intermediate features of the non-first diffusion module; and use a temporal attention unit to denoise the intermediate features of the non-first diffusion module to obtain the output features of the non-first diffusion module.

[0236] In one possible implementation of this disclosure, a spatial attention unit is used to denoise the input features using a deterministic denoising strategy and a spatial attention mechanism; and / or, a temporal attention unit is used to denoise the input features using a deterministic denoising strategy and a temporal attention mechanism.

[0237] In one possible implementation of this disclosure, in response to the length of the action posture sequence being greater than a set length threshold, the posture guidance features include multiple posture guidance sub-features. The extraction module 830 is used to: split the action posture sequence to obtain multiple action posture sub-sequences; wherein, adjacent action posture sub-sequences contain overlapping elements; and extract the posture guidance sub-features of the multiple action posture sub-sequences respectively.

[0238] In one possible implementation of this disclosure, the set length threshold is determined based on the input requirements of the video generation model.

[0239] In one possible implementation of this disclosure, the generation module 840 is configured to: input any pose guidance sub-feature, noise feature, and appearance feature into a video generation model, and perform feature processing using multiple diffusion modules in the video generation model; use the output feature of the last diffusion module as the target sub-feature corresponding to any pose guidance sub-feature; and fuse the target sub-features corresponding to multiple pose guidance sub-features to obtain the target feature.

[0240] In one possible implementation of this disclosure, the generation module 840 is configured to: input any pose guidance sub-feature, noise feature, and appearance feature into a video generation model to perform feature processing using multiple diffusion modules in the video generation model; and use the output feature of the last diffusion module as the target feature.

[0241] In one possible implementation of this disclosure, the decoding module 850 is configured to: decode multiple target features to obtain multiple video segments; and synthesize the multiple video segments to obtain an action-driven video displaying a target object.

[0242] In one possible implementation of this disclosure, each diffusion module includes an attitude modulation unit, a spatial attention unit, and a temporal attention unit.

[0243] Attitude modulation unit, used to perform attitude modulation on input features;

[0244] Spatial attention unit, used to denoise input features based on spatial attention mechanism;

[0245] Temporal attention unit, used to denoise input features based on temporal attention mechanism.

[0246] In one possible implementation of this disclosure, the attitude modulation unit is used to perform attitude modulation on partial features in the attitude guidance sub-features of adjacent action attitude sub-sequences using the same initialization noise; wherein, the partial features are obtained by encoding overlapping elements.

[0247] In one possible implementation of this disclosure, a spatial attention unit and / or a temporal attention unit are used to smooth the features corresponding to overlapping elements during the denoising time step.

[0248] In one possible implementation of this disclosure, the noise features are generated using the following modules:

[0249] The sampling module is used to randomly sample from a Gaussian distribution to obtain noise characteristics.

[0250] The video generation apparatus of this disclosure automatically generates motion-driven videos of the target object by inputting a reference image displaying the target object, a sequence of motion postures (or a sequence of human key points) and noise features. The motion postures of the target object in the motion-driven video are consistent with the motion postures indicated by the motion posture sequence, which can improve the realism of the target object in the motion-driven video, that is, improve the video generation quality and enhance the user experience in digital human-driven scenarios.

[0251] With the above Figure 7 Corresponding to the training method of the video generation model provided in the embodiments, this disclosure also provides a training device for the video generation model. Since the training device for the video generation model provided in the embodiments of this disclosure is similar to the one described above... Figure 7 The training method for the video generation model provided in the embodiments corresponds to the training method for the video generation model provided in the embodiments of this disclosure. Therefore, the implementation method for the training method of the video generation model is also applicable to the training device for the video generation model provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0252] Figure 9 This is a schematic diagram of the structure of the training device for the video generation model provided in Embodiment 8 of this disclosure.

[0253] like Figure 9 As shown, the training device 900 for the video generation model may include: an acquisition module 910, an encoding and noise-adding module 920, a feature extraction module 930, a generation module 940, and a training module 950.

[0254] The acquisition module 910 is used to acquire the action video and the corresponding action posture sequence, as well as to acquire the reference image displaying the target object; wherein, the elements in the action posture sequence represent the action posture generated based on the human body key points in the video frames of the action video.

[0255] The encoding and noise-adding module 920 is used to encode the action video to obtain the first video feature, and add noise to the first video feature to obtain the second video feature;

[0256] The feature extraction module 930 is used to encode the action posture sequence to obtain posture guidance features, and to encode the reference image to obtain the appearance features of the target object;

[0257] The generation module 940 is used to generate predictive noise using a video generation model based on second video features, pose guidance features, and appearance features;

[0258] Training module 950 is used to train the video generation model based on the difference between added noise and predicted noise in the first video features.

[0259] The training apparatus for the video generation model in this embodiment can improve the prediction accuracy of the video generation model by performing supervised training, thereby improving the quality of the video output by the video generation model and enhancing the user experience in digital human-driven scenarios.

[0260] To implement the above embodiments, this disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the video generation method or the training method of the video generation model proposed in any of the above embodiments of this disclosure.

[0261] To implement the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the video generation method or the training method of the video generation model proposed in any of the above embodiments of this disclosure.

[0262] To implement the above embodiments, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the video generation method or the training method for the video generation model proposed in any of the above embodiments of this disclosure.

[0263] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0264] Figure 10 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown.

[0265] The electronic device may include the server and client components described in the above embodiments. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0266] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in ROM (Read-Only Memory) 1002 or loaded from storage unit 1007 into RAM (Random Access Memory) 1003. The RAM 1003 may also store various programs and data required for the operation of the device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An I / O (Input / Output) interface 1005 is also connected to the bus 1004.

[0267] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0268] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the video generation method or the video generation model training method described above. For example, in some embodiments, the video generation method or the video generation model training method described above can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by computing unit 1001, one or more steps of the video generation method or video generation model training method described above can be performed. Alternatively, in other embodiments, computing unit 1001 can be configured to perform the video generation method or video generation model training method described above by any other suitable means (e.g., by means of firmware).

[0269] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0270] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0271] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0272] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0273] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0274] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0275] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0276] According to the technical solution of this disclosure, by inputting a reference image displaying a target object, a sequence of motion postures (or a sequence of human key points) and noise features, a motion-driven video of the target object is automatically generated. In this motion-driven video, the motion posture of the target object is consistent with the motion posture indicated by the motion posture sequence, which can improve the realism of the target object in the motion-driven video, that is, improve the video generation quality and improve the user experience in digital human-driven scenarios.

[0277] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0278] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video generation method, comprising: Acquire a sequence of motion postures and display a reference image of a target object; wherein, the elements in the sequence of motion postures represent motion postures generated based on human key points; The reference image is encoded to obtain the appearance features of the target object; The posture guidance features are extracted from the action posture sequence, and the posture guidance features, noise features and appearance features are input into the video generation model to perform feature processing using multiple diffusion modules in the video generation model. The target features are determined based on the output features of the last diffusion module. The target features are decoded to obtain a motion-driven video showing the target object, wherein the motion posture of the target object in the motion-driven video is consistent with the motion posture indicated by the motion posture sequence.

2. The method according to claim 1, wherein, Each of the diffusion modules includes an attitude modulation unit, a spatial attention unit, and a temporal attention unit. In response to the fact that the length of the action posture sequence is not greater than a set length threshold, the first diffusion module among the plurality of diffusion modules is used to: An attitude modulation unit is used to perform attitude modulation on the noise features based on the attitude guidance features to obtain the modulation features of the first diffusion module; The modulation feature of the first diffusion module is spliced ​​with the appearance feature to obtain the spliced ​​feature of the first diffusion module; The splicing features of the first diffusion module are denoised using a spatial attention unit to obtain the intermediate features of the first diffusion module; The intermediate features of the first diffusion module are denoised using a temporal attention unit to obtain the output features of the first diffusion module.

3. The method according to claim 2, wherein, The non-first diffusion module among the plurality of diffusion modules is used for: An attitude modulation unit is used to perform attitude modulation on the output features of the previous diffusion module based on the attitude guidance features to obtain the modulation features of the non-first diffusion module; The modulation features of the non-first diffusion module are spliced ​​together with the appearance features to obtain the spliced ​​features of the non-first diffusion module. Spatial attention units are used to denoise the splicing features of the non-first diffusion modules to obtain the intermediate features of the non-first diffusion modules; The intermediate features of the non-first diffusion module are denoised using a temporal attention unit to obtain the output features of the non-first diffusion module.

4. The method according to claim 2 or 3, wherein, The spatial attention unit is used to denoise the input features using a deterministic denoising strategy and a spatial attention mechanism; And / or, The temporal attention unit is used to denoise the input features using a deterministic denoising strategy and a temporal attention mechanism.

5. The method according to claim 1, wherein, In response to the length of the action posture sequence being greater than a set length threshold, the posture guidance feature includes multiple posture guidance sub-features; Extracting posture guidance features from the action posture sequence includes: The action posture sequence is split into multiple action posture sub-sequences; wherein, adjacent action posture sub-sequences contain overlapping elements; The posture guidance features of the multiple action posture sub-sequences are extracted respectively.

6. The method according to claim 2 or 5, wherein, The set length threshold is determined based on the input requirements of the video generation model.

7. The method according to claim 5, wherein, The step of inputting the pose guidance features, the noise features, and the appearance features into the video generation model, and then performing feature processing using multiple diffusion modules in the video generation model, includes: The pose guidance sub-feature, the noise feature, and the appearance feature are input into the video generation model to perform feature processing using multiple diffusion modules in the video generation model; Determining the target feature based on the output feature of the last diffusion module includes: The output feature of the last diffusion module is used as the target sub-feature corresponding to any of the pose guidance sub-features; The target features are obtained by fusing the target sub-features corresponding to the multiple pose guidance sub-features.

8. The method according to claim 5, wherein, The step of inputting the pose guidance features, the noise features, and the appearance features into the video generation model, and then performing feature processing using multiple diffusion modules in the video generation model, includes: The pose guidance sub-feature, the noise feature, and the appearance feature are input into the video generation model to perform feature processing using multiple diffusion modules in the video generation model; Determining the target feature based on the output feature of the last diffusion module includes: The output feature of the last diffusion module is used as the target feature.

9. The method according to claim 8, wherein, Decoding the target features to obtain an action-driven video displaying the target object includes: Decoding multiple target features yields multiple video segments; The multiple video clips are combined to obtain an action-driven video displaying the target object.

10. The method according to claim 7 or 8, wherein, Each of the diffusion modules includes an attitude modulation unit, a spatial attention unit, and a temporal attention unit. The attitude modulation unit is used to perform attitude modulation on the input features; The spatial attention unit is used to denoise the input features based on the spatial attention mechanism; The temporal attention unit is used to denoise the input features based on the temporal attention mechanism.

11. The method according to claim 10, wherein, The attitude modulation unit is used to perform attitude modulation on a portion of the attitude guidance sub-features of the adjacent action attitude sub-sequences using the same initialization noise; The aforementioned features are obtained by encoding the overlapping elements.

12. The method according to claim 10, wherein, The spatial attention unit and / or the temporal attention unit are used to smooth the features corresponding to the overlapping elements during the denoising time step.

13. The method according to claim 1, wherein, The noise characteristics are generated using the following steps: The noise characteristics are obtained by randomly sampling the Gaussian distribution.

14. A training method for a video generation model, comprising: Acquire an action video and the corresponding action posture sequence, as well as acquire a reference image displaying the target object; wherein, the elements in the action posture sequence represent action postures generated based on human body key points in the video frames of the action video; The action video is encoded to obtain a first video feature, and noise is added to the first video feature to obtain a second video feature; The action posture sequence is encoded to obtain posture guidance features, and the reference image is encoded to obtain the appearance features of the target object; A video generation model is used to generate predictive noise based on the second video features, the pose guidance features, and the appearance features; The video generation model is trained based on the difference between the added noise of the first video feature and the predicted noise, and the video generation model is used to implement the video generation method as described in any one of claims 1-13.

15. A video generation apparatus, comprising: The acquisition module is used to acquire a sequence of action postures and display a reference image of a target object; wherein, the elements in the sequence of action postures represent action postures generated based on human key points; An encoding module is used to encode the reference image to obtain the appearance features of the target object; The extraction module is used to extract posture guidance features from the action posture sequence; The generation module is used to input the pose guidance features, noise features and appearance features into the video generation model, so as to perform feature processing using multiple diffusion modules in the video generation model; The target features are determined based on the output features of the last diffusion module. A decoding module is used to decode the target features to obtain a motion-driven video displaying the target object, wherein the motion posture of the target object in the motion-driven video is consistent with the motion posture indicated by the motion posture sequence.

16. The apparatus according to claim 15, wherein, Each of the diffusion modules includes an attitude modulation unit, a spatial attention unit, and a temporal attention unit. In response to the fact that the length of the action posture sequence is not greater than a set length threshold, the first diffusion module among the plurality of diffusion modules is used to: An attitude modulation unit is used to perform attitude modulation on the noise features based on the attitude guidance features to obtain the modulation features of the first diffusion module; The modulation feature of the first diffusion module is spliced ​​with the appearance feature to obtain the spliced ​​feature of the first diffusion module; The splicing features of the first diffusion module are denoised using a spatial attention unit to obtain the intermediate features of the first diffusion module; The intermediate features of the first diffusion module are denoised using a temporal attention unit to obtain the output features of the first diffusion module.

17. The apparatus according to claim 16, wherein, The non-first diffusion module among the plurality of diffusion modules is used for: An attitude modulation unit is used to perform attitude modulation on the output features of the previous diffusion module based on the attitude guidance features to obtain the modulation features of the non-first diffusion module; The modulation features of the non-first diffusion module are spliced ​​together with the appearance features to obtain the spliced ​​features of the non-first diffusion module. Spatial attention units are used to denoise the splicing features of the non-first diffusion modules to obtain the intermediate features of the non-first diffusion modules; The intermediate features of the non-first diffusion module are denoised using a temporal attention unit to obtain the output features of the non-first diffusion module.

18. The apparatus according to claim 16 or 17, wherein, The spatial attention unit is used to denoise the input features using a deterministic denoising strategy and a spatial attention mechanism; And / or, The temporal attention unit is used to denoise the input features using a deterministic denoising strategy and a temporal attention mechanism.

19. The apparatus according to claim 15, wherein, In response to the length of the action posture sequence being greater than a set length threshold, the posture guidance feature includes multiple posture guidance sub-features; The extraction module is used for: The action posture sequence is split into multiple action posture sub-sequences; wherein, adjacent action posture sub-sequences contain overlapping elements; The posture guidance features of the multiple action posture sub-sequences are extracted respectively.

20. The apparatus according to claim 16 or 19, wherein, The set length threshold is determined based on the input requirements of the video generation model.

21. The apparatus according to claim 19, wherein, The generation module is used for: The pose guidance sub-feature, the noise feature, and the appearance feature are input into the video generation model to perform feature processing using multiple diffusion modules in the video generation model; The output feature of the last diffusion module is used as the target sub-feature corresponding to any of the pose guidance sub-features; The target features are obtained by fusing the target sub-features corresponding to the multiple pose guidance sub-features.

22. The apparatus according to claim 19, wherein, The generation module is configured to include: The pose guidance sub-feature, the noise feature, and the appearance feature are input into the video generation model to perform feature processing using multiple diffusion modules in the video generation model; The output feature of the last diffusion module is used as the target feature.

23. The apparatus according to claim 22, wherein, The decoding module is used for: Decoding multiple target features yields multiple video segments; The multiple video clips are combined to obtain an action-driven video displaying the target object.

24. The apparatus according to claim 21 or 22, wherein, Each of the diffusion modules includes an attitude modulation unit, a spatial attention unit, and a temporal attention unit. The attitude modulation unit is used to perform attitude modulation on the input features; The spatial attention unit is used to denoise the input features based on the spatial attention mechanism; The temporal attention unit is used to denoise the input features based on the temporal attention mechanism.

25. The apparatus according to claim 24, wherein, The attitude modulation unit is used to perform attitude modulation on a portion of the attitude guidance sub-features of the adjacent action attitude sub-sequences using the same initialization noise; The aforementioned features are obtained by encoding the overlapping elements.

26. The apparatus according to claim 24, wherein, The spatial attention unit and / or the temporal attention unit are used to smooth the features corresponding to the overlapping elements during the denoising time step.

27. The apparatus according to claim 15, wherein, The noise characteristics are generated using the following modules: The sampling module is used to randomly sample the Gaussian distribution to obtain the noise features.

28. A training device for a video generation model, comprising: The acquisition module is used to acquire an action video and the corresponding action posture sequence, as well as to acquire a reference image displaying the target object; wherein, the elements in the action posture sequence represent the action postures generated based on human body key points in the video frames of the action video. An encoding and noise-adding module is used to encode the action video to obtain a first video feature, and to add noise to the first video feature to obtain a second video feature; The feature extraction module is used to encode the action posture sequence to obtain posture guidance features, and to encode the reference image to obtain the appearance features of the target object; The generation module is used to generate predictive noise using a video generation model based on the second video features, the pose guidance features, and the appearance features; A training module is used to train the video generation model based on the difference between the added noise of the first video feature and the predicted noise, the video generation model being used to implement the video generation method as described in any one of claims 1-13.

29. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the video generation method of any one of claims 1-13, or to perform the training method of the video generation model of claim 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the video generation method according to any one of claims 1-13, or to execute the training method of the video generation model according to claim 14.

31. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the video generation method according to any one of claims 1-13, or, when executed, implements the steps of the training method for the video generation model according to claim 14.

Images