Video generation method and device, video generation model training method and device, equipment and medium

Abstract

The invention provides a video generation method, a video generation model training method, a video generation model training device, equipment and a medium, relates to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes of AIGC, digital humans, intelligent e-commerce and the like. According to the specific implementation scheme, an action posture sequence and a reference image displaying a target object are obtained; wherein elements in the action posture sequence represent action postures generated according to the key points of the human body; encoding the reference image to obtain appearance characteristics of the target object; extracting attitude guidance features from the action attitude sequence, and generating target features according to the attitude guidance features, the noise features and the appearance features; and decoding the target feature to obtain an action driving video displaying the target object.

Description

Technical Field

[0001] The present disclosure relates to the field of AI (Artificial Intelligence), specifically to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC (Artificial Intelligence Generated Content), digital humans, and smart e-commerce, and in particular to methods, devices, equipment, and media for training video generation and video generation models. Background Art

[0002] Digital human motion driving technology is used to apply the motion posture of the driving object to the digital human (such as the target object), so that the digital human in the generated motion-driven video has the same body movements, gestures, facial expressions and lip movements as the driving object. This high-precision motion posture migration not only enhances the realism and interactivity of the digital human, but also greatly broadens its application scenarios. For example, digital human motion driving technology can be applied to the fields of live broadcasting, social networking, customer service, and gaming, and promote the rapid development of related industries. Summary of the invention

[0003] The present disclosure provides a method, apparatus, device and medium for video generation and training of a video generation model.

[0004] According to one aspect of the present disclosure, there is provided a video generation method, comprising:

[0005] Acquire an action posture sequence and a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of the human body;

[0006] Encoding the reference image to obtain appearance features of the target object;

[0007] Extracting posture guidance features from the action posture sequence, and generating target features according to the posture guidance features, noise features and the appearance features;

[0008] The target features are decoded to obtain an action-driven video showing the target object.

[0009] According to another aspect of the present disclosure, a method for training a video generation model is provided, comprising:

[0010] Acquire an action video and a corresponding action posture sequence, and acquire a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of a human body in a video frame of the action video;

[0011] Encoding the action video to obtain a first video feature, and adding noise to the first video feature to obtain a second video feature;

[0012] Encoding the action posture sequence to obtain posture guidance features, and encoding the reference image to obtain appearance features of the target object;

[0013] Generate prediction noise using a video generation model according to the second video feature, the posture guidance feature, and the appearance feature;

[0014] The video generation model is trained based on the difference between the added noise of the first video feature and the predicted noise.

[0015] According to another aspect of the present disclosure, there is provided a video generating device, comprising:

[0016] An acquisition module, used to acquire an action posture sequence and a reference image showing a target object; wherein the elements in the action posture sequence represent the action posture generated according to the key points of the human body;

[0017] An encoding module, used to encode the reference image to obtain the appearance features of the target object;

[0018] An extraction module, used for extracting posture guidance features from the action posture sequence;

[0019] A generating module, used for generating a target feature according to the posture guidance feature, the noise feature and the appearance feature;

[0020] The decoding module is used to decode the target feature to obtain an action-driven video showing the target object.

[0021] According to another aspect of the present disclosure, a training device for a video generation model is provided, comprising:

[0022] An acquisition module, used to acquire an action video and a corresponding action posture sequence, and acquire a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of a human body in a video frame of the action video;

[0023] An encoding and noise adding module, used for encoding the action video to obtain a first video feature, and adding noise to the first video feature to obtain a second video feature;

[0024] A feature extraction module is used to encode the action posture sequence to obtain posture guidance features, and to encode the reference image to obtain appearance features of the target object;

[0025] A generation module, configured to generate prediction noise according to the second video feature, the posture guidance feature and the appearance feature by using a video generation model;

[0026] A training module is used to train the video generation model according to the difference between the added noise of the first video feature and the predicted noise.

[0027] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0028] at least one processor; and

[0029] a memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video generation method proposed in the above-mentioned first aspect of the present disclosure, or execute the training method of the video generation model proposed in the above-mentioned second aspect of the present disclosure.

[0031] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium of computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the video generation method proposed in the above-mentioned first aspect of the present disclosure, or to execute the video generation model training method proposed in the above-mentioned other aspect of the present disclosure.

[0032] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the video generation method proposed in the above-mentioned first aspect of the present disclosure, or, when executed, implements the video generation model training method proposed in the above-mentioned second aspect of the present disclosure.

[0033] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0035] Figure 1 A schematic diagram of a flow chart of a video generation method provided in Embodiment 1 of the present disclosure;

[0036] Figure 2 A schematic diagram of the flow of a video generation method provided in Embodiment 2 of the present disclosure;

[0037] Figure 3A schematic diagram of the model structure and video generation principle of the video generation model provided in the embodiment of the present disclosure;

[0038] Figure 4 A schematic diagram of the flow of a video generation method provided in Embodiment 3 of the present disclosure;

[0039] Figure 5 A schematic diagram of the flow of a video generation method provided in Embodiment 4 of the present disclosure;

[0040] Figure 6 A flowchart of a video generation method provided in Embodiment 5 of the present disclosure;

[0041] Figure 7 A flowchart of a method for training a video generation model provided in Embodiment 6 of the present disclosure;

[0042] Figure 8 This is a schematic diagram of the structure of a video generating device provided in Embodiment 7 of the present disclosure;

[0043] Fig. 9 A schematic diagram of the structure of a training device for a video generation model provided in Embodiment 8 of the present disclosure;

[0044] Fig.10 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0045] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0046] With the rapid development of AIGC technology, digital humans, as a new type of human-computer interaction, are attracting more and more attention. At present, the digital human motion driving technology mainly focuses on lip movements, while body movements come from the recorded base plate, which results in the inability to match the digital human's voice, tone and image movements when driving the digital human, which damages the authenticity of the digital human.

[0047] Among them, digital humans can be divided into 2D (two-dimensional) digital humans and 3D (three-dimensional) digital humans. 2D digital humans have many advantages such as low cost and simple production.

[0048] In related technologies, high-quality 2D digital humans are mainly realized through deep learning technology. When driving the digital human, the main focus is on the lip shape driving of the digital human. For the motion driving of the digital human, it can be achieved based on the intelligent agent. By inputting the 2D key points of the human body and using the stable video diffusion model as a pre-training model, a video showing the motion driving of the digital human is generated.

[0049] However, the above models are mainly aimed at general intelligent agent scenarios, and the general generation quality and realism are average. Hand artifacts can be clearly seen, and the video timing is unstable, making them unusable in digital human scenarios.

[0050] Therefore, in response to at least one of the above-mentioned problems, the present disclosure proposes a method, device, equipment and medium for video generation and training of a video generation model.

[0051] The following describes the video generation and video generation model training method, device, equipment and medium of the embodiment of the present disclosure with reference to the accompanying drawings. Before describing the embodiment of the present disclosure in detail, for ease of understanding, the commonly used technical terms are first introduced:

[0052] A large model refers to a machine learning model with large-scale parameters and complex computational structures. It is usually built from deep neural networks and has billions or even hundreds of billions of parameters. The purpose of designing a large model is to improve the model's expressiveness and predictive performance, and to be able to handle more complex tasks and data. Large models are widely used in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems.

[0053] Figure 1 The figure is a flowchart of the video generation method provided in the first embodiment of the present disclosure.

[0054] The embodiment of the present disclosure takes the video generating method configured in a video generating device as an example. The video generating device can be applied to any electronic device so that the electronic device can perform a video generating function.

[0055] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0056] like Figure 1 As shown, the video generation method may include the following steps S101 to S104:

[0057] Step S101, obtaining an action posture sequence and a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of the human body.

[0058] The action posture series may also be referred to as a key point sequence or a human key point sequence, and the elements in the action posture sequence are used to characterize the action posture generated according to the human key points. For example, the human key points may be 2D key points, which are easier to obtain and more robust.

[0059] The target object can be any user object.

[0060] In the embodiments of the present disclosure, there is no restriction on the method of obtaining the motion posture sequence and the reference image. For example, the motion posture sequence and the reference image may be provided manually by relevant personnel, or the motion posture sequence and the reference image may be collected online, such as by using web crawler technology, or the motion posture sequence and the reference image may be artificially synthesized, etc. The embodiments of the present disclosure do not impose any restriction on this.

[0061] Step S102: Encode the reference image to obtain appearance features of the target object.

[0062] Among them, the appearance feature is used to indicate the texture detail information of the target object.

[0063] In the disclosed embodiment, the appearance of the target object in the reference image may be encoded based on the feature encoding technology to obtain the appearance features of the target object.

[0064] As an example, a reference image may be input into an encoder in a pre-trained VAE (Variational AutoEncoder), and the encoder in the VAE may be used to encode the reference image to obtain appearance features of the latent space.

[0065] Among them, VAE is a generative model that combines the ideas of autoencoder and variational Bayesian inference. It can learn the latent representation of input data and generate new and similar data samples based on the latent representation. VAE includes an encoder and a decoder. The encoder in VAE can map the input data to a point in the latent space, that is, the latent representation, and the decoder in VAE can generate new and similar data samples based on the latent representation.

[0066] Step S103, extracting posture guidance features from the action posture sequence, and generating target features according to the posture guidance features, noise features and appearance features.

[0067] There is no restriction on the method of acquiring noise features. For example, noise features can be collected online, or noise features can be artificially synthesized, or a set distribution (such as Gaussian distribution, uniform distribution, Poisson distribution, mixed distribution, etc.) can be randomly sampled to obtain noise features, and so on.

[0068] It should be noted that since the Gaussian distribution has good mathematical properties, is easy to implement, and has smooth characteristics, random sampling of the Gaussian distribution can obtain noise features, which can make the noise features have the characteristics of independent and identical distribution, which helps the model learn the continuity and smooth changes of the data, improves the quality of video generation, and helps the model better capture the statistical characteristics of the data and avoid overfitting.

[0069] In the disclosed embodiment, the posture guidance features may be extracted from the action posture sequence based on the feature extraction technology.

[0070] As an example, the encoder in the VAE can be used to encode the action pose sequence to obtain the pose guidance feature. Alternatively, the action pose sequence can be encoded based on the pose encoder to obtain the pose guidance feature. Alternatively, the action pose sequence can be feature extracted based on the pose guider network to obtain the pose guidance feature.

[0071] In the disclosed embodiment, the posture guidance feature, the noise feature and the appearance feature can be integrated to generate the target feature. Exemplarily, the deep learning technology can be used to fuse the posture guidance feature, the noise feature and the appearance feature to obtain the target feature.

[0072] Step S104: decode the target features to obtain an action-driven video showing the target object.

[0073] In the disclosed embodiment, the target features may be decoded based on feature decoding technology to obtain an action-driven video showing the target object.

[0074] As an example, the decoder in the VAE can be used to decode the target features to obtain an action-driven video. The action-driven video shows the target object, and the action posture of the target object is consistent with the action posture indicated by the action posture sequence, so that the video of the target object action can be driven to be generated according to the given action posture sequence.

[0075] The video generation method of the disclosed embodiment automatically generates an action-driven video of the target object by inputting a reference image showing the target object, an action posture sequence (or called a human body key point sequence) and noise features, wherein the action posture of the target object in the action-driven video is consistent with the action posture indicated by the action posture sequence, which can improve the authenticity of the target object in the action-driven video, that is, improve the quality of video generation, and improve the user experience in the digital human driving scenario.

[0076] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, and are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0077] In order to clearly illustrate how to generate target features based on posture guidance features, noise features and appearance features in any embodiment of the present disclosure, the present disclosure also proposes a video generation method.

[0078] Figure 2 This is a flowchart of the video generation method provided in the second embodiment of the present disclosure.

[0079] like Figure 2 As shown, the video generation method may include the following steps S201 to S206:

[0080] Step S201, obtaining an action posture sequence and a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of the human body.

[0081] Step S202: Encode the reference image to obtain appearance features of the target object.

[0082] Step S203: extracting posture guidance features from the action posture sequence.

[0083] For explanations of steps S201 to S203 , reference may be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0084] Step S204, inputting the posture guidance features, the noise features and the appearance features into the video generation model, so as to perform feature processing using a plurality of diffusion modules in the video generation model.

[0085] As a possible implementation method, when the length of the action posture sequence is not greater than the set length threshold, the posture guidance features, noise features and appearance features can be directly input into the video generation model at the same time, so as to use multiple serial diffusion modules in the video generation model for feature processing.

[0086] The set length threshold may be a pre-set length threshold. Exemplarily, the set length threshold may be set according to input requirements of the video generation model.

[0087] In any embodiment of the present disclosure, each diffusion module may include a pose modulation (PoseModulation) unit, a spatial attention (Spatial Self-Attention) unit and a temporal attention (TemporalSelf-Attention) unit, wherein the pose modulation unit is used to perform pose modulation on input features; the spatial attention unit is used to denoise the input features based on a spatial attention mechanism (or a spatial self-attention mechanism); and the temporal attention unit is used to denoise the input features based on a temporal attention mechanism (or a temporal self-attention mechanism).

[0088] As an example, when the length of the action posture sequence is not greater than the set length threshold, the first diffusion module among the plurality of diffusion modules connected in series is used to perform the following steps A to D:

[0089] Step A: adopting a posture modulation unit to perform posture modulation on the noise feature based on the posture guidance feature to obtain the modulation feature of the first diffusion module, so that the modulation feature can effectively inject the key point information of the human body.

[0090] Among them, the dimensions of the noise features are consistent with those of the posture guidance features.

[0091] Step B: Concatenate the modulation features and appearance features of the first diffusion module to obtain the concatenated features of the first diffusion module.

[0092] As an example, the modulation features and appearance features of the first diffusion module can be spliced ​​in the token (token refers to the smallest processing unit of data, which can refer to an image block, a pixel group or a feature vector) dimension to obtain the spliced ​​features of the first diffusion module.

[0093] Step C: Use the spatial attention unit to denoise the concatenated features of the first diffusion module to obtain the intermediate features of the first diffusion module.

[0094] Among them, the spatial attention unit (or spatial attention layer) is used to learn how to migrate the reference image to the specified action posture sequence.

[0095] Exemplarily, the spatial attention unit can denoise the concatenated features of the first diffusion module based on the spatial attention mechanism to obtain the intermediate features of the first diffusion module.

[0096] Step D: Use the temporal attention unit to denoise the intermediate features of the first diffusion module to obtain the output features of the first diffusion module.

[0097] In the present disclosure, in order to ensure the temporal stability of video generation, the continuity of the video can be improved through the spatial self-attention mechanism.

[0098] Exemplarily, the temporal attention unit may denoise the intermediate features of the first diffusion module based on the temporal attention mechanism to obtain the output features of the first diffusion module.

[0099] As an example, a diffusion module other than the first one (such as the i-th diffusion module, where i is a positive integer greater than 1) among the plurality of diffusion modules connected in series is used to perform the following steps A' to D':

[0100] Step A': adopting a posture modulation unit to perform posture modulation on the output feature of the i-1th diffusion module based on the posture guidance feature to obtain the modulation feature of the i-th diffusion module.

[0101] Step B': splicing the modulation feature and the appearance feature of the i-th diffusion module to obtain the splicing feature of the i-th diffusion module. The implementation principle is similar to step B and will not be described here.

[0102] Step C': Use the spatial attention unit to denoise the spliced ​​features of the i-th diffusion module to obtain the intermediate features of the i-th diffusion module. The implementation principle is similar to step C and will not be repeated here.

[0103] Step D': Use the temporal attention unit to denoise the intermediate features of the i-th diffusion module to obtain the output features of the i-th diffusion module. The implementation principle is similar to step D and will not be repeated here.

[0104] In summary, the use of posture modulation, spatial attention mechanism, and temporal attention mechanism for feature processing can improve the temporal stability, continuity, and generation quality of video generation, and further enhance the user experience in digital human-driven scenarios.

[0105] In any embodiment of the present disclosure, the spatial attention unit in each diffusion module can be used to denoise the input features using a deterministic denoising strategy and a spatial attention mechanism.

[0106] In any embodiment of the present disclosure, the temporal attention unit in each diffusion module can be used to denoise the input features using a deterministic denoising strategy and a temporal attention mechanism.

[0107] In any embodiment of the present disclosure, the spatial attention unit in each diffusion module can be used to denoise the input features by adopting a deterministic denoising strategy and a spatial attention mechanism, and the temporal attention unit in each diffusion module can also be used to denoise the input features by adopting a deterministic denoising strategy and a temporal attention mechanism.

[0108] Therefore, using a deterministic denoising strategy to gradually denoise the features can reduce the disturbance of the denoising process and further improve the quality of video generation.

[0109] Step S205: determining the target feature according to the output feature of the last diffusion module.

[0110] As a possible implementation manner, when the length of the action posture sequence is not greater than a set length threshold, the output feature of the last diffusion module among the multiple diffusion modules connected in series may be used as the target feature.

[0111] Step S206: decode the target features to obtain an action-driven video showing the target object.

[0112] For the explanation of step S206, reference may be made to the relevant description in any embodiment of the present disclosure, and no further details will be given here.

[0113] As an example, the model structure of the video generation model and the video generation principle can be as follows: Figure 3 As shown in FIG. 1 , the video generation model includes N DiTs (Diffusion Models with Transformers) connected in series. Each DiT module may include a pose modulation unit, a spatial self-attention unit, and a temporal self-attention unit. Figure 3 Reshape in this case means changing the shape of the feature tensor to adapt to the input requirements of the next network layer.

[0114] The principle of video generation is as follows: the reference image is input into the encoder in the VAE for encoding to obtain the appearance features, the action-driven sequence is input into the pose guidance (Pose Guider) network for feature extraction to obtain the pose guidance features, and the pose modulation (Pose Modulation) unit in the first DiT module is used to perform pose modulation on the noise features based on the pose guidance features to obtain the modulation features. After that, the modulation features are concatenated with the appearance features, and are successively input into the series-connected spatial attention (Spatial Self-Attention) unit and temporal attention (Temporal Self-Attention) unit for step-by-step denoising to obtain the output features of the first DiT module.

[0115] The output features of the first DiT module are used to replace the noise features and input into the second DiT module. The pose modulation unit in the second DiT module performs pose modulation on the output features of the first DiT module based on the pose guidance features to obtain the modulation features. After that, the modulation features are concatenated with the appearance features and input into the series-connected spatial self-attention unit and temporal self-attention unit in sequence for step-by-step denoising to obtain the output features of the second DiT module.

[0116] By analogy, the output features of the Nth DiT module can be obtained, and the output features of the Nth DiT module can be decoded by the decoder in the VAE to obtain the action-driven video.

[0117] The video generation method of the disclosed embodiment adopts multiple diffusion modules to gradually denoise the input features. Each diffusion module can focus on removing a part of the noise, and can achieve layer-by-layer refinement of the generated results, and gradually improve the quality of the generated video. That is, each denoising step can be regarded as an optimization of the result of the previous step. This progressive optimization process helps to generate more detailed and realistic videos and enhance the stability and generalization ability of the model.

[0118] In order to clearly illustrate the above embodiment, the present disclosure further proposes a video generating method.

[0119] Figure 4 This is a flowchart of the video generation method provided in the third embodiment of the present disclosure.

[0120] like Figure 4 As shown, the video generation method may include the following steps S401 to S406:

[0121] Step S401, obtaining an action posture sequence and a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of the human body.

[0122] Step S402: Encode the reference image to obtain appearance features of the target object.

[0123] For explanations of steps S401 to S402, reference may be made to the relevant descriptions in any embodiment of the present disclosure, and will not be repeated here.

[0124] Step S403 , in response to the length of the action posture sequence being greater than a set length threshold, splitting the action posture sequence to obtain a plurality of action posture sub-sequences; wherein adjacent action posture sub-sequences contain overlapping elements.

[0125] The set length threshold may be a pre-set length threshold. Exemplarily, the set length threshold may be set according to input requirements of the video generation model.

[0126] It should be noted that, based on the input requirements of the video generation model, setting the set length threshold has at least the following advantages: reducing memory consumption (reducing the memory requirement for each forward and backward propagation); improving computing efficiency (supporting parallel processing and batch processing, speeding up training and inference); avoiding gradient vanishing and explosion, and improving the stability of the model through local gradient calculation; simplifying model design (converting long sequences into subsequences of fixed length can simplify model design and implementation); improving model generalization ability and helping the model learn local features; flexible attention mechanism (more effective application of local and global attention mechanisms).

[0127] In an embodiment of the present disclosure, when the length of a motion posture sequence is greater than a set length threshold, the motion sequence can be split based on the set length threshold to obtain multiple motion posture sub-sequences, wherein the length of each motion posture sub-sequence is not greater than the set length threshold, and adjacent motion posture sub-sequences contain overlapping elements.

[0128] Step S404: extracting posture guidance sub-features of a plurality of action posture sub-sequences respectively.

[0129] In the disclosed embodiment, the posture guidance sub-features of each action posture sub-sequence may be extracted based on feature extraction technology.

[0130] As an example, the encoder in the VAE can be used to encode each action posture subsequence to obtain the posture guidance sub-feature. Alternatively, each action posture subsequence can be encoded based on the posture encoder to obtain the posture guidance sub-feature. Alternatively, each action posture subsequence can be feature extracted based on the pose guider network to obtain the pose guidance sub-feature.

[0131] Step S405: Generate target features according to the multiple posture guidance sub-features, noise features and appearance features.

[0132] In the disclosed embodiment, deep learning technology may be used to integrate the posture guidance sub-features, noise features, and appearance features to generate target features.

[0133] Step S406: decode the target features to obtain an action-driven video showing the target object.

[0134] For the explanation of step S406, reference may be made to the relevant description in any embodiment of the present disclosure, and will not be repeated here.

[0135] The video generation method of the disclosed embodiment, when the action posture sequence is long, divides the action posture sequence, and the divided subsequences can be processed in parallel, which can improve the generation efficiency of the video. In addition, the divided subsequences can help the model to better learn local features, and these local features can be combined to form global features, thereby improving the generalization ability and robustness of the model. In addition, the overlapping elements between adjacent subsequences can ensure the smooth transition of the generated actions between different subsequences, avoid abrupt jumps or faults, and further improve the generation quality of the video to generate a more refined and realistic video.

[0136] In order to clearly explain how the target features are generated based on multiple posture guidance sub-features, noise features and appearance features in the above embodiments of the present disclosure, the present disclosure also proposes a video generation method.

[0137] Figure 5 This is a flowchart of the video generation method provided in Embodiment 4 of the present disclosure.

[0138] like Figure 5 As shown, the video generation method may include the following steps S501 to S508:

[0139] Step S501, obtaining an action posture sequence and a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of the human body.

[0140] Step S502: Encode the reference image to obtain appearance features of the target object.

[0141] Step S503 , in response to the length of the action posture sequence being greater than a set length threshold, splitting the action posture sequence to obtain a plurality of action posture sub-sequences.

[0142] Among them, adjacent action posture subsequences contain overlapping elements.

[0143] Step S504: extracting posture guidance sub-features of a plurality of action posture sub-sequences respectively.

[0144] For explanations of steps S501 to S504, reference may be made to the relevant descriptions in any embodiment of the present disclosure, and will not be repeated here.

[0145] Step S505: input any posture guidance sub-feature, noise feature and appearance feature into the video generation model, so as to perform feature processing using multiple diffusion modules in the video generation model.

[0146] In any embodiment of the present disclosure, each diffusion module may include a pose modulation (PoseModulation) unit, a spatial attention (Spatial Self-Attention) unit and a temporal attention (TemporalSelf-Attention) unit, wherein the pose modulation unit is used to perform pose modulation on input features, the spatial attention unit is used to denoise the input features based on the spatial attention mechanism, and the temporal attention unit is used to denoise the input features based on the temporal attention mechanism.

[0147] As an example, the first diffusion module among the plurality of diffusion modules connected in series is used to perform the following steps E to H:

[0148] Step E: For any one of the multiple posture guidance sub-features, a posture modulation unit is used to perform posture modulation on the noise feature based on the posture guidance sub-feature to obtain a modulation feature of the first diffusion module, so that the modulation feature can effectively inject key point information of the human body.

[0149] Among them, the dimensions of the noise feature and the attitude guidance sub-feature are consistent.

[0150] Step F: splicing the modulation feature and the appearance feature of the first diffusion module to obtain the splicing feature of the first diffusion module.

[0151] Step G: The spatial attention unit is used to denoise the concatenated features of the first diffusion module based on the spatial attention mechanism to obtain the intermediate features of the first diffusion module.

[0152] Among them, the spatial attention unit (or spatial attention layer) is used to learn how to migrate the reference image to the specified action posture sequence.

[0153] Step H: Use the temporal attention unit to denoise the intermediate features of the first diffusion module based on the temporal attention mechanism to obtain the output features of the first diffusion module.

[0154] In the present disclosure, in order to ensure the temporal stability of video generation, the continuity of the video can be improved through the spatial self-attention mechanism.

[0155] As an example, a diffusion module other than the first one (such as the i-th diffusion module, where i is a positive integer greater than 1) among the plurality of diffusion modules connected in series is used to perform the following steps E' to H':

[0156] Step E': adopting a posture modulation unit to perform posture modulation on the output feature of the i-1th diffusion module based on the posture guidance sub-feature, so as to obtain the modulation feature of the i-th diffusion module.

[0157] Step F': splicing the modulation feature and the appearance feature of the i-th diffusion module to obtain the splicing feature of the i-th diffusion module. The implementation principle is similar to that of step F and will not be described in detail here.

[0158] Step G': The spatial attention unit is used to denoise the concatenated features of the i-th diffusion module based on the spatial attention mechanism to obtain the intermediate features of the i-th diffusion module.

[0159] Step H': Use the temporal attention unit to denoise the intermediate features of the i-th diffusion module based on the temporal attention mechanism to obtain the output features of the i-th diffusion module.

[0160] In summary, the use of posture modulation, spatial attention mechanism, and temporal attention mechanism for feature processing can improve the temporal stability, continuity, and generation quality of video generation, and further enhance the user experience in digital human-driven scenarios.

[0161] In any embodiment of the present disclosure, the posture modulation unit in each diffusion module is used to use the same initialization noise to perform posture modulation on some features in the posture guidance sub-features of adjacent action posture sub-sequences; wherein the partial features are obtained by encoding overlapping elements.

[0162] Therefore, by using the same initialization noise to process the features of the overlapping parts, a smooth transition between adjacent subsequences can be ensured, abrupt changes in the generated results at the boundaries can be avoided, and the overall coherence and contextual consistency of the generated video can be improved. In addition, by maintaining the consistency of the features of the overlapping parts, the detailed information in the video can be better preserved, making the generated video more detailed, realistic and coherent.

[0163] In any of the embodiments of the present disclosure, the spatial attention unit and / or the temporal attention unit in each diffusion module is also used to smooth the features corresponding to the overlapping elements in the denoising time step.

[0164] In summary, smoothing can ensure that the feature changes between adjacent subsequences are smoother, avoid abrupt changes or discontinuities at the boundaries of the generated results, maintain the continuity of the features in time, and make the generated video smoother and more natural. In addition, smoothing helps to retain the detailed information of the overlapping parts, avoid the loss of details due to noise or discontinuity, and better restore the texture and structural information in the video, making the generated video more detailed and realistic. In addition, smoothing can enhance the alignment of overlapping features, ensure that these features are consistent in different subsequences, and improve the signal-to-noise ratio of the features, so that the model can better focus on the real features of the data.

[0165] Step S506: Use the output feature of the last diffusion module as the target sub-feature corresponding to any posture guidance sub-feature.

[0166] In the disclosed embodiment, the output feature of the last diffusion module among the plurality of diffusion modules connected in series may be used as the target sub-feature corresponding to the above-mentioned posture guidance sub-feature.

[0167] Step S507: fuse the target sub-features corresponding to the multiple posture guidance features to obtain the target feature.

[0168] In the disclosed embodiment, target sub-features corresponding to a plurality of posture guidance features may be fused based on feature fusion technology to obtain target features.

[0169] Step S508: decode the target features to obtain an action-driven video showing the target object.

[0170] For explanation of step S508, reference may be made to the relevant description in any embodiment of the present disclosure, and will not be repeated here.

[0171] The video generation method of the disclosed embodiment adopts multiple diffusion modules to gradually denoise the input features. Each diffusion module can focus on removing a part of the noise, and can achieve layer-by-layer refinement of the generated results, and gradually improve the quality of the generated video. That is, each denoising step can be regarded as an optimization of the result of the previous step. This progressive optimization process helps to generate more detailed and realistic videos and enhance the stability and generalization ability of the model.

[0172] In order to clearly explain how the target features are generated based on multiple posture guidance sub-features, noise features and appearance features in the above embodiments of the present disclosure, the present disclosure also proposes a video generation method.

[0173] Figure 6 This is a flowchart of the video generation method provided in Embodiment 5 of the present disclosure.

[0174] like Figure 6 As shown, the video generation method may include the following steps S601 to S608:

[0175] Step S601, obtaining an action posture sequence and a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of the human body.

[0176] Step S602: Encode the reference image to obtain appearance features of the target object.

[0177] Step S603 , in response to the length of the action posture sequence being greater than a set length threshold, splitting the action posture sequence to obtain a plurality of action posture sub-sequences.

[0178] Among them, adjacent action posture subsequences contain overlapping elements.

[0179] Step S604: extracting posture guidance sub-features of a plurality of action posture sub-sequences respectively.

[0180] For explanations of steps S601 to S604, reference may be made to the relevant descriptions in any embodiment of the present disclosure, and will not be repeated here.

[0181] Step S605: input any posture guidance sub-feature, noise feature and appearance feature into the video generation model, so as to perform feature processing using multiple diffusion modules in the video generation model.

[0182] The implementation principle of step S605 is similar to that of step S505, and will not be described in detail here.

[0183] Step S606: Use the output feature of the last diffusion module as the target feature.

[0184] In the disclosed embodiment, the output feature of the last diffusion module among the plurality of diffusion modules connected in series may be directly used as the target feature.

[0185] Step S607: decode multiple target features to obtain multiple video clips.

[0186] In the disclosed embodiment, multiple target features may be decoded separately to obtain multiple video clips.

[0187] As an example, the decoder in the VAE can be used to decode each target feature to obtain a video clip.

[0188] Step S608: synthesize the multiple video clips to obtain an action-driven video showing the target object.

[0189] In the disclosed embodiment, multiple video clips can be synthesized based on video synthesis technology to obtain an action-driven video showing a target object. The action posture of the target object in the action-driven video is consistent with the action posture indicated by the action posture sequence, so that a video of the target object action can be driven to be generated according to a given action posture sequence.

[0190] The video generation method of the disclosed embodiment adopts multiple diffusion modules to process multiple posture guidance sub-features respectively, which can better retain the detail information in each subsequence, making the generated video more refined and realistic. In addition, each diffusion module can focus on removing a part of the noise, and can achieve layer-by-layer refinement of the generation result, and gradually improve the quality of the generated video. That is, each denoising step can be regarded as an optimization of the result of the previous step. This progressive optimization process helps to generate more refined and realistic videos and enhance the stability and generalization ability of the model.

[0191] By decoding multiple target features separately, the detail information in each subsequence can be better preserved, making the generated video clips more refined and realistic. By synthesizing multiple video clips, it can ensure that the feature changes between adjacent subsequences are smoother, avoiding abrupt changes at the boundaries of the generated action-driven video, making the synthesized action-driven video more continuous in time and the transition of action more natural and smooth.

[0192] The above are various embodiments corresponding to the application method of the video generation model (ie, the video generation method). The present disclosure also proposes a training method for the video generation model.

[0193] Figure 7 A flowchart of a method for training a video generation model provided in Embodiment 6 of the present disclosure.

[0194] like Figure 7 As shown, the training method of the video generation model may include the following steps S701 to S705:

[0195] Step S701, obtaining an action video and a corresponding action posture sequence, and obtaining a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of a human body in a video frame of the action video.

[0196] The target object can be any user object.

[0197] In the embodiments of the present disclosure, there is no limitation on the method of obtaining the reference image. For example, the reference image can be obtained from a training set or a test set, or the reference image can be provided manually by a user, or the reference image can be collected online, or an image of the target object can be collected to obtain the reference image, or the reference image can be artificially synthesized, etc. The embodiments of the present disclosure do not impose any limitation on this.

[0198] In the embodiments of the present disclosure, there is no limitation on the method of obtaining the action video. For example, the action video can be obtained from a training set or a test set, or the action video can be collected online, or the driving object can be filmed to obtain the action video, or the action video can be artificially synthesized, etc. The embodiments of the present disclosure do not limit this either.

[0199] The driving object and the target object in the action video may be the same object, or may be different objects, and the embodiments of the present disclosure do not impose any limitation on this.

[0200] Exemplarily, when the driving object and the target object are the same object, the reference image may be a certain video frame in the action video.

[0201] In the disclosed embodiments, key point detection can be performed on the driving object in each frame of the action video to obtain the human body key point information of the driving object in each frame of the video, and based on the human body key point information of the driving object in each frame of the video, a motion posture sequence is generated; wherein the elements in the motion posture sequence are used to characterize the motion posture generated according to the human body key points in the video frames.

[0202] Step S702: Encode the action video to obtain a first video feature, and add noise to the first video feature to obtain a second video feature.

[0203] In the embodiment of the present disclosure, the action video can be encoded based on the feature encoding technology to obtain the video latent space feature, which is recorded as the first video feature in the present disclosure. Exemplarily, the encoder in the pre-trained VAE can be used to encode the action video to obtain the first video feature.

[0204] In the disclosed embodiment, noise may also be added to the first video feature to obtain the second video feature.

[0205] Step S703, encoding the action posture sequence to obtain posture guidance features, and encoding the reference image to obtain appearance features of the target object.

[0206] Among them, the appearance feature is used to indicate the texture detail information of the target object.

[0207] In the disclosed embodiment, the posture guidance features may be extracted from the action posture sequence based on the feature extraction technology.

[0208] As an example, the encoder in the VAE can be used to encode the action pose sequence to obtain the pose guidance feature. Alternatively, the action pose sequence can be encoded based on the pose encoder to obtain the pose guidance feature. Alternatively, the action pose sequence can be feature extracted based on the pose guider network to obtain the pose guidance feature.

[0209] In the disclosed embodiment, the appearance of the target object in the reference image may be encoded based on the feature encoding technology to obtain the appearance features of the target object.

[0210] As an example, a reference image can be input into an encoder in a pre-trained VAE, and the encoder in the VAE is used to encode the reference image to obtain the appearance features of the latent space.

[0211] Step S704: using a video generation model to generate predicted noise according to the second video feature, the posture guidance feature and the appearance feature.

[0212] In the disclosed embodiment, the second video feature, the posture guidance feature, the appearance feature and the noise step (for indicating the degree of noise addition) can be input into the video generation model together for noise prediction to obtain the predicted noise output by the video generation model.

[0213] Step S705: training a video generation model according to the difference between the added noise and the predicted noise of the first video feature.

[0214] In an embodiment of the present disclosure, the value of the loss function (referred to as the loss value in the present disclosure) can be determined based on the difference between the added noise and the predicted noise of the first video feature. Thus, in the present disclosure, the model parameters in the video generation model can be adjusted based on the loss value to minimize the loss value.

[0215] Among them, the loss value is positively correlated with the above difference, that is, the greater the difference, the greater the loss value, and conversely, the smaller the difference, the smaller the loss value.

[0216] It should be noted that the above only uses the termination condition of model training as the minimization of the loss value as an example. In actual application, other termination conditions can also be set. For example, the termination conditions can also include: the training time reaches the set time, the training rounds reach the set rounds, and so on.

[0217] As a possible implementation, Figure 7 The video generation model trained by the embodiment shown can be used to implement Figures 1 to 6The video generation method described in any embodiment.

[0218] The training method of the video generation model of the embodiment of the present disclosure can improve the prediction accuracy of the video generation model by performing supervised training on the video generation model, that is, improve the video quality output by the video generation model, and improve the user experience in the digital human driving scenario.

[0219] In any one of the embodiments of the present disclosure, taking the video generation model as an example of a Transformer-based diffusion model, a frame of reference image and a sequence of action postures (i.e., a sequence of 2D key points of the human body) can be input into the diffusion model to generate a high-quality action-driven video using the diffusion model, thereby achieving matching of action and voice and improving the naturalness of the digital human.

[0220] In the model training stage: given an action video and a corresponding action posture sequence (i.e., a 2D key point sequence of the human body), the action video is input into the encoder in the pre-trained VAE for encoding to obtain the first video feature, and the first video feature is added with noise to the second video feature. At the same time, the action posture sequence is input into the encoder in the VAE for encoding to obtain a posture guidance feature (or key point feature) with the same dimension as the first video feature, and the encoder in the VAE is used to extract texture details from a given reference image showing a target object to obtain the appearance feature of the target object. Afterwards, the second video feature, posture guidance feature, appearance feature, and noise addition step are input into the diffusion model together, and the added noise is predicted by the diffusion model, and the diffusion model is trained based on the difference between the predicted noise and the added noise of the first video feature.

[0221] The diffusion model is based on the Transformer structure and contains N DiT modules. Each DiT module mainly consists of three parts: pose modulation unit, spatial self-attention unit and temporal self-attention unit. The noise is predicted through N DiT modules with the same structure, and the loss value is calculated based on the predicted noise and the actual noise, so as to update the diffusion model based on the loss value.

[0222] Among them, the posture modulation unit uses the posture guidance feature to modulate the noise feature to obtain the modulation feature; among them, the dimensions of the posture guidance feature and the noise feature are consistent, which can effectively inject the key point information of the human body. Then, the modulation feature and the appearance feature are spliced ​​in the token dimension and input into the spatial self-attention unit. At this time, the different video frames of the action video are in the batch dimension. Through this layer, we learn how to migrate the reference image to the specified action posture sequence. In order to ensure the temporal stability of video generation, the same position features of different video frames can be input into the temporal self-attention unit, and the continuity of the generated video can be improved through the spatial self-attention mechanism.

[0223] In the model reasoning stage or test stage: given the action posture sequence to be generated and the reference image showing the target object, the noise is randomly sampled in the Gaussian distribution to obtain the noise feature, and the final action-driven video is output through iterative multi-step denoising. Since the length of each output video is limited, in order to improve the action continuity of the long video, multiple strategies can be used to improve the video stability during the denoising process. First, the long action posture sequence is divided into overlapping subsequences. During the video generation process, the same action frame uses the same initialization noise to improve consistency. At the same time, a deterministic denoising process (such as DDIM (Denoising Diffusion Implicit Models, denoising diffusion implicit model)) is used to reduce the disturbance of the denoising process. In the denoising time step, the continuity of the video is further improved by smoothing the overlapping area. Finally, the output target feature is input into the decoder in the VAE for decoding, and finally a long-term stable digital human-driven video is obtained, which is recorded as an action-driven video in this disclosure.

[0224] In summary, posture modulation, spatial attention mechanism, temporal attention mechanism, etc. are used to improve the video generation effect. Moreover, a series of smoothing and stabilizing denoising strategies are used to improve the continuity of the generated video. High-quality digital human motion-driven videos can be generated, which improves the user experience in video generation scenarios.

[0225] With the above Figures 1 to 6 Corresponding to the video generation method provided in the embodiment, the present disclosure also provides a video generation device. Figures 1 to 6 The video generation method provided in the embodiment corresponds to the embodiment, so the implementation of the video generation method is also applicable to the video generation device provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0226] Figure 8 This is a structural diagram of a video generating device provided in Embodiment 7 of the present disclosure.

[0227] like Figure 8 As shown, the video generating device 800 may include: an acquisition module 810 , an encoding module 820 , an extraction module 830 , a generation module 840 and a decoding module 850 .

[0228] The acquisition module 810 is used to acquire a motion posture sequence and a reference image showing a target object; wherein the elements in the motion posture sequence represent the motion posture generated according to the key points of the human body;

[0229] The encoding module 820 is used to encode the reference image to obtain the appearance characteristics of the target object;

[0230] An extraction module 830, for extracting posture guidance features from the action posture sequence;

[0231] A generating module 840, for generating a target feature according to the posture guidance feature, the noise feature and the appearance feature;

[0232] The decoding module 850 is used to decode the target features to obtain an action-driven video showing the target object.

[0233] In a possible implementation of the embodiment of the present disclosure, the generation module 840 is used to: input posture guidance features, noise features and appearance features into a video generation model to perform feature processing using multiple diffusion modules in the video generation model; and determine the target features based on the output features of the last diffusion module.

[0234] In a possible implementation of the embodiment of the present disclosure, each diffusion module includes a posture modulation unit, a spatial attention unit and a temporal attention unit. In response to the length of the action posture sequence being not greater than a set length threshold, the first diffusion module among the multiple diffusion modules is used to: use the posture modulation unit to perform posture modulation on the noise feature based on the posture guidance feature to obtain the modulation feature of the first diffusion module; splice the modulation feature of the first diffusion module with the appearance feature to obtain the splicing feature of the first diffusion module; use the spatial attention unit to denoise the splicing feature of the first diffusion module to obtain the intermediate feature of the first diffusion module; use the temporal attention unit to denoise the intermediate feature of the first diffusion module to obtain the output feature of the first diffusion module.

[0235] In a possible implementation of the embodiment of the present disclosure, a non-first diffusion module among multiple diffusion modules is used to: use a posture modulation unit to perform posture modulation on the output features of the previous diffusion module based on the posture guidance features to obtain the modulation features of the non-first diffusion module; splice the modulation features of the non-first diffusion module with the appearance features to obtain the spliced ​​features of the non-first diffusion module; use a spatial attention unit to denoise the spliced ​​features of the non-first diffusion module to obtain the intermediate features of the non-first diffusion module; use a temporal attention unit to denoise the intermediate features of the non-first diffusion module to obtain the output features of the non-first diffusion module.

[0236] In a possible implementation of the embodiments of the present disclosure, a spatial attention unit is used to denoise input features using a deterministic denoising strategy and a spatial attention mechanism; and / or a temporal attention unit is used to denoise input features using a deterministic denoising strategy and a temporal attention mechanism.

[0237] In a possible implementation of the embodiments of the present disclosure, in response to the length of the action posture sequence being greater than a set length threshold, the posture guidance feature includes multiple posture guidance sub-features, and the extraction module 830 is used to: split the action posture sequence to obtain multiple action posture sub-sequences; wherein adjacent action posture sub-sequences contain overlapping elements; and extract the posture guidance sub-features of the multiple action posture sub-sequences respectively.

[0238] In a possible implementation of the embodiment of the present disclosure, the length threshold is set according to the input requirements of the video generation model.

[0239] In a possible implementation of the embodiments of the present disclosure, the generation module 840 is used to: input any posture guidance sub-feature, noise feature and appearance feature into a video generation model, so as to perform feature processing using multiple diffusion modules in the video generation model; use the output feature of the last diffusion module as the target sub-feature corresponding to any posture guidance sub-feature; and fuse the target sub-features corresponding to multiple posture guidance sub-features to obtain the target feature.

[0240] In a possible implementation of the embodiment of the present disclosure, the generation module 840 is used to include: inputting any posture guidance sub-feature, noise feature and appearance feature into the video generation model to use multiple diffusion modules in the video generation model for feature processing; and using the output feature of the last diffusion module as the target feature.

[0241] In a possible implementation of the embodiment of the present disclosure, the decoding module 850 is used to: decode multiple target features to obtain multiple video clips; synthesize the multiple video clips to obtain an action-driven video showing the target object.

[0242] In a possible implementation of the embodiment of the present disclosure, each diffusion module includes a posture modulation unit, a spatial attention unit and a temporal attention unit.

[0243] A posture modulation unit, used for performing posture modulation on the input feature;

[0244] Spatial attention unit, used to denoise input features based on the spatial attention mechanism;

[0245] Temporal attention unit, used to denoise input features based on the temporal attention mechanism.

[0246] In a possible implementation of the embodiment of the present disclosure, a posture modulation unit is used to perform posture modulation on some features in the posture guidance sub-features of adjacent action posture sub-sequences using the same initialization noise; wherein the some features are obtained by encoding overlapping elements.

[0247] In a possible implementation of the embodiments of the present disclosure, the spatial attention unit and / or the temporal attention unit is used to smooth features corresponding to overlapping elements in a denoising time step.

[0248] In a possible implementation of the embodiment of the present disclosure, the noise feature is generated by using the following modules:

[0249] The sampling module is used to randomly sample the Gaussian distribution to obtain noise characteristics.

[0250] The video generation device of the disclosed embodiment automatically generates an action-driven video of the target object by inputting a reference image showing the target object, an action posture sequence (or called a human body key point sequence) and a noise feature, wherein the action posture of the target object in the action-driven video is consistent with the action posture indicated by the action posture sequence, which can improve the authenticity of the target object in the action-driven video, that is, improve the quality of video generation, and improve the user experience in the digital human driving scenario.

[0251] With the above Figure 7 Corresponding to the training method of the video generation model provided in the embodiment, the present disclosure also provides a training device for the video generation model. Figure 7 The training method of the video generation model provided in the embodiment corresponds to the embodiment, so the implementation method of the video generation model training method is also applicable to the training device of the video generation model provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0252] Fig. 9 This is a structural diagram of a training device for a video generation model provided in Embodiment 8 of the present disclosure.

[0253] like Fig. 9 As shown, the training device 900 of the video generation model may include: an acquisition module 910, a coding and denoising module 920, a feature extraction module 930, a generation module 940 and a training module 950.

[0254] The acquisition module 910 is used to acquire an action video and a corresponding action posture sequence, and to acquire a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of a human body in a video frame of the action video;

[0255] The coding and noise adding module 920 is used to encode the action video to obtain a first video feature, and add noise to the first video feature to obtain a second video feature;

[0256] The feature extraction module 930 is used to encode the action posture sequence to obtain the posture guidance feature, and encode the reference image to obtain the appearance feature of the target object;

[0257] A generation module 940, configured to generate prediction noise according to the second video feature, the posture guidance feature and the appearance feature using a video generation model;

[0258] The training module 950 is used to train the video generation model according to the difference between the added noise and the predicted noise of the first video feature.

[0259] The training device of the video generation model of the embodiment of the present disclosure can improve the prediction accuracy of the video generation model by performing supervised training on the video generation model, that is, improve the video quality output by the video generation model, and improve the user experience in the digital human driving scenario.

[0260] In order to implement the above embodiments, the present disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video generation method or the video generation model training method proposed in any of the above embodiments of the present disclosure.

[0261] In order to implement the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the video generation method or the video generation model training method proposed in any of the above embodiments of the present disclosure.

[0262] In order to implement the above embodiments, the present disclosure also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the video generation method or the video generation model training method proposed in any of the above embodiments of the present disclosure.

[0263] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0264] Fig.10 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown.

[0265] Among them, the electronic device may include the server and the client in the above-mentioned embodiment. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0266] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1002 or a computer program loaded from a storage unit 1007 to a RAM (Random Access Memory) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An I / O (Input / Output) interface 1005 is also connected to the bus 1004.

[0267] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0268] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the above-mentioned video generation method or the training method of the video generation model. For example, in some embodiments, the above-mentioned video generation method or the training method of the video generation model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the video generation method or the video generation model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the above-described video generation method or the video generation model training method by any other appropriate means (e.g., by means of firmware).

[0269] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0270] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0271] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0272] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0273] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0274] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0275] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0276] According to the technical solution of the embodiment of the present disclosure, a reference image showing the target object, a motion posture sequence (or called a human body key point sequence) and noise features are input to automatically generate an action-driven video of the target object, wherein the action posture of the target object in the action-driven video is consistent with the action posture indicated by the action posture sequence, which can improve the authenticity of the target object in the action-driven video, that is, improve the quality of video generation, and improve the user experience in the digital human driving scenario.

[0277] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0278] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video generation method, comprising: Acquire an action posture sequence and a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of the human body; Encoding the reference image to obtain appearance features of the target object; Extracting posture guidance features from the action posture sequence, and generating target features according to the posture guidance features, noise features and the appearance features; The target features are decoded to obtain an action-driven video showing the target object.

2. The method according to claim 1, wherein: The step of generating a target feature according to the posture guidance feature, the noise feature and the appearance feature comprises: Inputting the posture guidance feature, the noise feature and the appearance feature into a video generation model to perform feature processing using a plurality of diffusion modules in the video generation model; The target feature is determined according to the output feature of the last diffusion module.

3. The method according to claim 2, wherein: Each of the diffusion modules includes a posture modulation unit, a spatial attention unit and a temporal attention unit, In response to the length of the action posture sequence being less than a set length threshold, a first diffusion module among the plurality of diffusion modules is configured to: Using a posture modulation unit to perform posture modulation on the noise feature based on the posture guidance feature to obtain a modulation feature of the first diffusion module; splicing the modulation feature of the first diffusion module with the appearance feature to obtain a splicing feature of the first diffusion module; Using a spatial attention unit to denoise the concatenated features of the first diffusion module to obtain intermediate features of the first diffusion module; A temporal attention unit is used to denoise the intermediate features of the first diffusion module to obtain output features of the first diffusion module.

4. The method according to claim 3, wherein: The non-first diffusion module among the plurality of diffusion modules is used for: Using a posture modulation unit to perform posture modulation on the output feature of the previous diffusion module based on the posture guidance feature to obtain the modulation feature of the non-first diffusion module; splicing the modulation feature of the non-first diffusion module with the appearance feature to obtain a splicing feature of the non-first diffusion module; Using a spatial attention unit to denoise the concatenated features of the non-first diffusion module to obtain the intermediate features of the non-first diffusion module; A temporal attention unit is used to denoise the intermediate features of the non-first diffusion module to obtain the output features of the non-first diffusion module.

5. The method according to claim 3 or 4, wherein: The spatial attention unit is used to denoise the input features by adopting a deterministic denoising strategy and a spatial attention mechanism; and / or, The temporal attention unit is used to denoise input features by adopting a deterministic denoising strategy and a temporal attention mechanism.

6. The method according to claim 2, wherein: In response to the length of the action posture sequence being greater than a set length threshold, the posture guidance feature includes a plurality of posture guidance sub-features; The step of extracting posture guidance features from the action posture sequence comprises: Splitting the action posture sequence to obtain a plurality of action posture subsequences; wherein adjacent action posture subsequences contain overlapping elements; The posture guidance sub-features of the plurality of action posture sub-sequences are respectively extracted.

7. The method according to claim 3 or 6, wherein: The set length threshold is determined according to the input requirement of the video generation model.

8. The method according to claim 6, wherein: The step of inputting the posture guidance feature, the noise feature and the appearance feature into a video generation model to perform feature processing using a plurality of diffusion modules in the video generation model comprises: Inputting any posture guidance sub-feature, the noise feature and the appearance feature into a video generation model to perform feature processing using multiple diffusion modules in the video generation model; Determining the target feature according to the output feature of the last diffusion module includes: The output feature of the last diffusion module is used as the target sub-feature corresponding to any posture guidance sub-feature; The target sub-features corresponding to the multiple posture guidance sub-features are fused to obtain the target feature.

9. The method according to claim 6, wherein: The step of inputting the posture guidance feature, the noise feature and the appearance feature into a video generation model to perform feature processing using a plurality of diffusion modules in the video generation model comprises: Inputting any posture guidance sub-feature, the noise feature and the appearance feature into a video generation model to perform feature processing using multiple diffusion modules in the video generation model; Determining the target feature according to the output feature of the last diffusion module includes: The last output feature of the diffusion module is used as the target feature.

10. The method according to claim 9, wherein: The decoding of the target feature to obtain an action-driven video showing the target object includes: Decoding a plurality of the target features to obtain a plurality of video clips; The multiple video clips are synthesized to obtain an action-driven video showing the target object.

11. The method according to claim 8 or 9, wherein: Each of the diffusion modules includes a posture modulation unit, a spatial attention unit and a temporal attention unit, The posture modulation unit is used to perform posture modulation on the input feature; The spatial attention unit is used to denoise the input features based on the spatial attention mechanism; The temporal attention unit is used to denoise input features based on the temporal attention mechanism.

12. The method according to claim 11, wherein: The posture modulation unit is used to perform posture modulation on some features of the posture guidance sub-features of the adjacent action posture sub-sequences by using the same initialization noise; The partial features are obtained by encoding the overlapping elements.

13. The method according to claim 11, wherein: The spatial attention unit and / or the temporal attention unit is used to smooth the features corresponding to the overlapping elements in the denoising time step.

14. The method according to claim 1, wherein: The noise signature is generated using the following steps: The Gaussian distribution is randomly sampled to obtain the noise feature.

15. A method for training a video generation model, comprising: Acquire an action video and a corresponding action posture sequence, and acquire a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of a human body in a video frame of the action video; Encoding the action video to obtain a first video feature, and adding noise to the first video feature to obtain a second video feature; Encoding the action posture sequence to obtain posture guidance features, and encoding the reference image to obtain appearance features of the target object; Generate prediction noise using a video generation model according to the second video feature, the posture guidance feature, and the appearance feature; The video generation model is trained based on the difference between the added noise of the first video feature and the predicted noise.

16. A video generating device, comprising: An acquisition module, used to acquire an action posture sequence and a reference image showing a target object; wherein the elements in the action posture sequence represent the action posture generated according to the key points of the human body; An encoding module, used to encode the reference image to obtain the appearance features of the target object; An extraction module, used for extracting posture guidance features from the action posture sequence; A generating module, used for generating a target feature according to the posture guidance feature, the noise feature and the appearance feature; The decoding module is used to decode the target feature to obtain an action-driven video showing the target object.

17. The device according to claim 16, wherein: The generating module is used for: Inputting the posture guidance feature, the noise feature and the appearance feature into a video generation model to perform feature processing using a plurality of diffusion modules in the video generation model; The target feature is determined according to the output feature of the last diffusion module.

18. The device according to claim 17, wherein: Each of the diffusion modules includes a posture modulation unit, a spatial attention unit and a temporal attention unit, In response to the length of the action posture sequence being less than a set length threshold, a first diffusion module among the plurality of diffusion modules is configured to: Using a posture modulation unit to perform posture modulation on the noise feature based on the posture guidance feature to obtain a modulation feature of the first diffusion module; splicing the modulation feature of the first diffusion module with the appearance feature to obtain a splicing feature of the first diffusion module; Using a spatial attention unit to denoise the concatenated features of the first diffusion module to obtain intermediate features of the first diffusion module; A temporal attention unit is used to denoise the intermediate features of the first diffusion module to obtain output features of the first diffusion module.

19. The device according to claim 18, wherein: The non-first diffusion module among the plurality of diffusion modules is used for: Using a posture modulation unit to perform posture modulation on the output feature of the previous diffusion module based on the posture guidance feature to obtain the modulation feature of the non-first diffusion module; splicing the modulation feature of the non-first diffusion module with the appearance feature to obtain a splicing feature of the non-first diffusion module; Using a spatial attention unit to denoise the concatenated features of the non-first diffusion module to obtain the intermediate features of the non-first diffusion module; A temporal attention unit is used to denoise the intermediate features of the non-first diffusion module to obtain the output features of the non-first diffusion module.

20. The device according to claim 18 or 19, wherein: The spatial attention unit is used to denoise the input features by adopting a deterministic denoising strategy and a spatial attention mechanism; and / or, The temporal attention unit is used to denoise input features by adopting a deterministic denoising strategy and a temporal attention mechanism.

21. The device according to claim 17, wherein: In response to the length of the action posture sequence being greater than a set length threshold, the posture guidance feature includes a plurality of posture guidance sub-features; The extraction module is used to: Splitting the action posture sequence to obtain a plurality of action posture subsequences; wherein adjacent action posture subsequences contain overlapping elements; The posture guidance sub-features of the plurality of action posture sub-sequences are respectively extracted.

22. The device according to claim 18 or 21, wherein: The set length threshold is determined according to the input requirement of the video generation model.

23. The device according to claim 21, wherein The generating module is used for: Inputting any posture guidance sub-feature, the noise feature and the appearance feature into a video generation model to perform feature processing using multiple diffusion modules in the video generation model; The output feature of the last diffusion module is used as the target sub-feature corresponding to any posture guidance sub-feature; The target sub-features corresponding to the multiple posture guidance sub-features are fused to obtain the target feature.

24. The device according to claim 21, wherein The generating module is used to include: Inputting any posture guidance sub-feature, the noise feature and the appearance feature into a video generation model to perform feature processing using multiple diffusion modules in the video generation model; The last output feature of the diffusion module is used as the target feature.

25. The device according to claim 24, wherein: The decoding module is used for: Decoding a plurality of the target features to obtain a plurality of video clips; The multiple video clips are synthesized to obtain an action-driven video showing the target object.

26. The device according to claim 23 or 24, wherein: Each of the diffusion modules includes a posture modulation unit, a spatial attention unit and a temporal attention unit, The posture modulation unit is used to perform posture modulation on the input feature; The spatial attention unit is used to denoise the input features based on the spatial attention mechanism; The temporal attention unit is used to denoise input features based on the temporal attention mechanism.

27. The device according to claim 26, wherein: The posture modulation unit is used to perform posture modulation on some features of the posture guidance sub-features of the adjacent action posture sub-sequences by using the same initialization noise; The partial features are obtained by encoding the overlapping elements.

28. The device according to claim 26, wherein The spatial attention unit and / or the temporal attention unit is used to smooth the features corresponding to the overlapping elements in the denoising time step.

29. The device according to claim 16, wherein: The noise signature is generated using the following modules: The sampling module is used to randomly sample the Gaussian distribution to obtain the noise characteristics.

30. A training device for a video generation model, comprising: An acquisition module, used to acquire an action video and a corresponding action posture sequence, and acquire a reference image showing a target object; wherein the elements in the action posture sequence represent action postures generated according to key points of a human body in a video frame of the action video; An encoding and noise adding module, used for encoding the action video to obtain a first video feature, and adding noise to the first video feature to obtain a second video feature; A feature extraction module is used to encode the action posture sequence to obtain posture guidance features, and to encode the reference image to obtain appearance features of the target object; A generation module, configured to generate prediction noise according to the second video feature, the posture guidance feature and the appearance feature by using a video generation model; A training module is used to train the video generation model according to the difference between the added noise of the first video feature and the predicted noise.

31. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video generation method described in any one of claims 1-14, or execute the training method of the video generation model described in claim 15.

32. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the video generation method according to any one of claims 1 to 14, or to execute the video generation model training method according to claim 15.

33. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the video generation method according to any one of claims 1 to 14, or, when executed, implements the steps of the video generation model training method according to claim 15.

Images