Video generation model training method and device, video generation method and device and electronic equipment

Abstract

The invention provides a training method of a video generation model, a video generation method and device, and electronic equipment, and the training method comprises the steps: carrying out the feature extraction of a reference image, and obtaining first multi-modal feature information; performing feature extraction on the driving video to obtain second multi-modal feature information; generating control condition feature information according to the first multi-mode feature information and the second multi-mode feature information; generating original noise according to the reference image and the driving video, and encoding the reference image to obtain an embedded vector of the reference image; performing noise prediction according to the control condition feature information, the original noise and the embedded vector of the reference image to obtain predicted noise conforming to the control condition feature information; and according to the difference between the original noise and the predicted noise, adjusting parameters of the video generation model to obtain a trained video generation model. Therefore, by using the multi-modal condition control information, the stability of the generation action can be ensured.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence fields such as machine learning, and in particular to the field of video generation, and more particularly to a video generation model training method, a video generation method, a device, and an electronic device. Background Art

[0002] In the field of video generation, controllable human motion video generation has always received extensive attention. In related technologies, a two-stage training strategy is usually used to learn human motion control and ensure temporal consistency between video frames. However, the above learning method using the two-stage training strategy often suffers from severe motion instability and cannot ensure the coexistence of appearance consistency and temporal stability. Summary of the invention

[0003] The embodiments of the present disclosure provide a video generation model training method, a video generation method, a device and an electronic device.

[0004] According to a first aspect of an embodiment of the present disclosure, a video generation model training method is proposed, comprising:

[0005] Performing feature extraction on a reference image to obtain first multimodal feature information of the reference image, wherein the first multimodal feature information includes depth feature information and bone feature information;

[0006] Extracting features from the driving video to obtain second multimodal feature information, where the second multimodal feature information includes depth feature information and skeleton feature information;

[0007] generating control condition feature information according to the first multimodal feature information and the second multimodal feature information;

[0008] generating original noise according to the reference image and the driving video;

[0009] Performing encoding processing on the reference image to obtain an embedding vector of the reference image;

[0010] Perform noise prediction according to the control condition feature information, the original noise and the embedded vector of the reference image to obtain predicted noise that meets the control condition feature information;

[0011] According to the difference between the original noise and the predicted noise, the parameters of the video generation model are adjusted to obtain a trained video generation model.

[0012] According to a second aspect of an embodiment of the present disclosure, a video generation method is proposed, including:

[0013] Obtain an image to be processed and obtain a reference action feature sequence;

[0014] Using a pre-trained video generation model, noise prediction is performed on the image to be processed and the action feature sequence to obtain predicted noise;

[0015] Based on the denoising module of the video generation model, multiple rounds of iterative denoising processing are performed using the predicted noise to obtain video generation feature information;

[0016] Decoding the video generation feature information using a VAE decoder of the video generation model to obtain a video;

[0017] Among them, the video generation model is a model trained using the training method described in the first aspect above.

[0018] According to a third aspect of an embodiment of the present disclosure, a video generation model training device is proposed, comprising:

[0019] A first feature extraction module, configured to extract features from a reference image to obtain first multimodal feature information of the reference image, wherein the first multimodal feature information includes depth feature information and bone feature information;

[0020] A second feature extraction module, used to extract features from the driving video to obtain second multimodal feature information, where the second multimodal feature information includes depth feature information and bone feature information;

[0021] A first generating module, configured to generate control condition feature information according to the first multimodal feature information and the second multimodal feature information;

[0022] A second generating module, used for generating original noise according to the reference image and the driving video;

[0023] An encoding module, used for encoding the reference image to obtain an embedding vector of the reference image;

[0024] A noise prediction module, used for performing noise prediction according to the control condition characteristic information, the original noise and the embedded vector of the reference image, to obtain predicted noise that meets the control condition characteristic information;

[0025] The parameter adjustment module is used to adjust the parameters of the video generation model according to the difference between the original noise and the predicted noise to obtain a trained video generation model.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a video generating device is provided, including:

[0027] An acquisition module is used to acquire an image to be processed and obtain a reference action feature sequence;

[0028] A noise prediction module, used for performing noise prediction on the image to be processed and the action feature sequence using a pre-trained video generation model to obtain predicted noise;

[0029] A denoising processing module, which is used to perform multiple rounds of iterative denoising processing based on the denoising module of the video generation model using the predicted noise to obtain video generation feature information;

[0030] A decoding processing module, used to decode the video generation feature information using the VAE decoder of the video generation model to obtain a video;

[0031] Among them, the video generation model is a model trained using the training method described in the first aspect above.

[0032] According to a fifth aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0033] at least one processor;

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect and the second aspect.

[0036] According to a sixth aspect of an embodiment of the present disclosure, a storage medium is proposed, wherein the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the method described in the first and second aspects above.

[0037] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is proposed, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method described in the first aspect and the second aspect are implemented.

[0038] According to the technical solution disclosed in the present invention, the result of generating a human body video according to multimodal condition control information can be achieved, and the controllability and smoothness are greatly enhanced, thereby ensuring the stability of the generated action. In addition, the disclosed embodiment uses multimodal motion guidance to accurately capture the appearance and motion details, and applies classifier-free guidance to ensure that the generated result is consistent with the latent space specified by the control condition, so that stable action video generation can be achieved even when the control condition is jittery, which can ensure the coexistence of appearance consistency and timing stability.

[0039] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 A flowchart of a video generation model training method provided by an embodiment of the present disclosure;

[0042] Figure 2 A flowchart of a video generation model training method provided by an embodiment of the present disclosure;

[0043] Figure 3a A schematic diagram of the structure of a feature extractor provided in an embodiment of the present disclosure;

[0044] Figure 3b A schematic diagram of the structure of a spatial attention unit or a temporal attention unit provided in an embodiment of the present disclosure;

[0045] Figure 4 A schematic diagram of an implementation flow of generating prediction noise provided by an embodiment of the present disclosure;

[0046] Figure 5 An example diagram of the structure of a mutual information adapter provided in an embodiment of the present disclosure;

[0047] Figure 6 A schematic diagram of a framework of a video generation model provided in an embodiment of the present disclosure;

[0048] Figure 7 A schematic diagram of a flow chart of a video generation method provided by an embodiment of the present disclosure;

[0049] Figure 8 A block diagram of a video generation model training device provided by an embodiment of the present disclosure;

[0050] Fig. 9 A block diagram of a video generation model training device provided by an embodiment of the present disclosure;

[0051] Fig.10 A block diagram of a video generation model training device provided by an embodiment of the present disclosure;

[0052] Fig.11 A block diagram of a video generating device provided by an embodiment of the present disclosure;

[0053] Fig.12 A block diagram of an electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0055] The embodiments of the present disclosure are not exhaustive, but are only illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined, for example, some or all of the steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0056] In each embodiment of the present disclosure, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment based on their internal logical relationships.

[0057] In the embodiments of the present disclosure, unless otherwise specified, elements expressed in the singular form, such as "a", "an", "the", "above", "said", "aforementioned", "this", etc., may mean "one and only one", or "one or more", "at least one", etc. For example, when using articles such as "a", "an", "the" in English in translation, the noun after the article may be understood as a singular expression or a plural expression.

[0058] In the embodiments of the present disclosure, “plurality” refers to two or more.

[0059] In some embodiments, the terms “at least one,” “one or more,” “a plurality of,” “multiple,” etc. may be used interchangeably.

[0060] The prefixes such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute restrictions on the position, order, priority, quantity or content of the description objects. The statement of the description object refers to the description in the context of the claims or embodiments, and should not constitute unnecessary restrictions due to the use of prefixes. For example, if the description object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and the "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the description object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of description objects is not limited by the ordinal number, and can be one or more. Taking the "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes may be the same or different. For example, if the description object is "device", then the "first device" and the "second device" may be the same device or different devices, and their types may be the same or different. For another example, if the description object is "information", then the "first information" and the "second information" may be the same information or different information, and their contents may be the same or different.

[0061] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0062] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0063] It is worth noting that in the embodiments of the present disclosure, certain software, components, models, etc. that already exist in the industry may be mentioned. They should be considered as exemplary and their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0064] The following describes the video generation model training method, video generation method, device and electronic device of the embodiments of the present disclosure with reference to the accompanying drawings.

[0065] Figure 1 The following is a flow chart of a video generation model training method provided by an embodiment of the present disclosure. Figure 1 As shown, the video generation model training method may include but is not limited to the following steps.

[0066] In step 101, feature extraction is performed on a reference image to obtain first multimodal feature information of the reference image.

[0067] In some embodiments, a driving video may be obtained, and a frame may be selected from the driving video as a reference image. Exemplarily, a frame may be randomly sampled from the driving video as a reference image. Exemplarily, the driving video may be a sequence of images of multiple frames randomly sampled from a video sample as the driving video, for example, a sequence of images of 25 frames may be randomly sampled as the driving video. After obtaining the reference image, feature extraction may be performed on the reference image to obtain first multimodal feature information of the reference image.

[0068] In some embodiments, the first multimodal feature information may include depth feature information and bone feature information. Exemplarily, the depth information can be extracted from the reference image, and the depth image of the reference image is generated according to the depth information and the reference image, and features are extracted from the depth image to obtain the depth feature information. The depth image can be subjected to bone extraction using a bone extraction algorithm to obtain bone feature information. The obtained depth feature information and bone feature information are used as the first multimodal feature information of the reference image. It can be understood that the above-mentioned implementation method for determining the depth feature information and the bone feature information is only an example given to facilitate the understanding of those skilled in the art, that is, the depth feature information and the bone feature information of the reference image can also be determined by other optional implementation methods, and the present disclosure does not limit this and will not repeat it.

[0069] In step 102, feature extraction is performed on the driving video to obtain second multimodal feature information.

[0070] Exemplarily, the driving video may be processed frame by frame. In one possible implementation, feature extraction may be performed on each frame of the driving video to obtain multimodal feature information (including depth feature information and bone feature information) of each frame, and the set of multimodal feature information of each frame of the driving video is used as the second multimodal feature information of the driving video.

[0071] In some embodiments, the second multimodal feature information may include depth feature information and bone feature information. Exemplarily, for any frame image in the driving video, depth information can be extracted from the frame image, a depth image of the frame image is generated according to the depth information and the frame image, and features are extracted from the depth image to obtain depth feature information. The depth image can be subjected to bone extraction using a bone extraction algorithm to obtain bone feature information. The obtained depth feature information and bone feature information are used as multimodal feature information of the frame image. The set of multimodal feature information of each frame image in the driving video is used as the second multimodal feature information of the driving video. It can be understood that the above-mentioned implementation method for determining the depth feature information and the bone feature information is only an example given to facilitate the understanding of those skilled in the art, that is, the depth feature information and the bone feature information of the frame image can also be determined by other optional implementation methods, and this disclosure does not limit this and will not be repeated.

[0072] It should be noted that, in some embodiments, the above step 101 and step 102 may be performed simultaneously, or step 101 may be performed first and then step 102, or step 102 may be performed first and then step 101.

[0073] In step 103, control condition feature information is generated according to the first multimodal feature information and the second multimodal feature information.

[0074] In some embodiments, feature extraction may be performed on the first multimodal feature information and the second multimodal feature information to obtain control condition feature information. Exemplarily, a feature extractor in a video generation model may be used to extract features from the first multimodal feature information and the second multimodal feature information to obtain control condition feature information. In one possible implementation, the feature extractor may be a spatial-temporal attention mechanism to implement feature extraction. Exemplarily, control condition features may be extracted from the first multimodal feature information and the second multimodal feature information respectively through a spatial-temporal attention mechanism, and finally the control condition feature information may be obtained.

[0075] It should be noted that the control condition characteristic information can be understood as the characteristic information of the control condition, and the control condition can refer to the input variables used to control the behavior of the video generation model, which directly affect the performance and stability of the model. Exemplarily, the skeleton (or skeleton) and the depth map can be used as control conditions. Exemplarily, the features of the skeleton (or skeleton) and the depth map are extracted, and the extracted skeleton features and depth features are multimodally fused, so that the control condition characteristic information can be obtained, that is, the characteristic information for the control condition.

[0076] In step 104, original noise is generated according to the reference image and the driving video.

[0077] In some embodiments, the driving video may be subjected to noise processing, and features of the noisy driving video may be combined with features of the reference image to obtain the above-mentioned original noise.

[0078] In one possible implementation, the video generation model includes a VAE (Variational Auto-Encoder), and the VAE encoder can be used to encode the reference image to obtain the first encoding feature information of the reference image; the VAE encoder can be used to encode the driving video to obtain the second encoding feature information of the driving video; the second encoding feature information is added with noise and then merged with the first encoding feature information to obtain the above-mentioned original noise. Exemplarily, the merging operation can refer to a splicing operation, in which the second encoding feature information is added with noise and then spliced ​​with the first encoding feature information to obtain the above-mentioned original noise. Exemplarily, the added noise can be but is not limited to Gaussian noise.

[0079] It should be noted that, since the driving video includes multiple frames of images and the reference image is a single frame of image, in order to ensure that the reference image has the same number of features as the driving video, after obtaining the coding features of the reference image, the coding features of the reference image can be copied according to the number of frames of the driving video, and the feature information obtained after the copying is used as the first coding feature information. Exemplarily, when the number of frames of the driving video is 25, when obtaining the coding features of the reference image, the coding features of the reference image are copied according to the number of frames of the driving video to obtain the coding features of 25 frames of reference images, and the set of coding features of the 25 frames of reference images is used as the first coding feature information.

[0080] In step 105, the reference image is encoded to obtain an embedding vector of the reference image.

[0081] In some embodiments, the video generation model may include a CLIP network, and the CLIP network may be used to encode the reference image to obtain an embedding vector of the reference image.

[0082] It should be noted that, in some embodiments, the above steps 103, 104 and 105 may be performed in an exchanged order or simultaneously.

[0083] In step 106, noise prediction is performed based on the control condition feature information, the original noise and the embedded vector of the reference image to obtain predicted noise that meets the control condition feature information.

[0084] It should be noted that, in some embodiments, the video generation model in this article uses SVD (Stable Video Diffusion, a potential video diffusion model) as the basic model, rather than optimizing on the SD (Stable Diffusion) image generation basic model, and utilizes the prior information of the video generation model.

[0085] In some embodiments, the video generation model may include a motion control module and a generation module, and the motion control module includes a conditional encoder. In some embodiments, the weight and structure of the generation module can be copied from the original SVD, and the generation module weights can be frozen, and the encoder weights can be copied from the original SVD to construct the conditional encoder. The encoder and conditional encoder of the generation module are used to encode the control condition feature information, the original noise, and the embedded vector of the reference image to obtain the encoding processing result, and the decoder of the generation module is used to decode the encoding processing result to obtain the predicted noise that meets the control condition feature information.

[0086] In step 107, the parameters of the video generation model are adjusted according to the difference between the original noise and the predicted noise to obtain a trained video generation model.

[0087] In some embodiments, a loss value is determined based on the difference between the original noise and the predicted noise, and the parameters of the video generation model are adjusted using a gradient descent method based on the loss value to obtain a trained video generation model.

[0088] In a possible implementation, the inter-frame difference of the original noise is obtained, and the inter-frame difference of the predicted noise is obtained; the inter-frame variance loss value is generated according to the inter-frame difference of the original noise and the inter-frame difference of the predicted noise; and the parameters of the video generation model are adjusted according to the inter-frame variance loss value. Thus, the consistency and continuity between the input video features and the latent features generated by the video generation model can be constrained by the inter-frame variance loss, which can further enhance the temporal stability.

[0089] Exemplarily, the inter-frame difference of the original noise can be obtained, and the inter-frame difference of the predicted noise can be obtained; the inter-frame variance loss value is generated according to the inter-frame difference of the original noise and the inter-frame difference of the predicted noise; the parameters of the video generation model are adjusted according to the inter-frame variance loss value, for example, the inter-frame variance loss value is used as the model loss value, and the parameters of the video generation model are updated using the model loss value. Exemplarily, the corresponding gradient information can be calculated using a gradient descent algorithm based on the inter-frame variance loss value, and the parameters of the video generation model can be updated based on the gradient information. For example, the value of each parameter to be adjusted in the video generation model can be updated based on the gradient information so that it moves in the opposite direction of the gradient, thereby reducing the model loss value; wherein the gradient information can represent the direction in which the loss function changes the fastest. It can be understood that other parameter adjustment methods can also be used to adjust the parameters of the video generation model, which is not limited in this disclosure and will not be repeated.

[0090] By implementing the disclosed embodiments, the result of generating a human body video according to multimodal condition control information can be achieved, and the controllability and smoothness are greatly enhanced, thereby ensuring the stability of the generated action. In addition, the disclosed embodiments use multimodal motion guidance to accurately capture the appearance and motion details, and apply classifier-free guidance (i.e., using unsupervised learning methods to automatically discover potential structures and patterns in the data) to ensure that the generated results are consistent with the latent space specified by the control conditions, so that stable action video generation can be achieved even when the control conditions are jittery, and the coexistence of appearance consistency and timing stability can be guaranteed.

[0091] Figure 2 The following is a flow chart of a video generation model training method provided by an embodiment of the present disclosure. Figure 2 As shown, the video generation model training method may include but is not limited to the following steps.

[0092] In step 201, feature extraction is performed on a reference image to obtain first multimodal feature information of the reference image.

[0093] Optional implementations of step 201 can be found in Figure 1 Optional implementations of step 101, and Figure 1 Other related parts in the embodiments involved will not be described in detail here.

[0094] In step 202, feature extraction is performed on the driving video to obtain second multimodal feature information.

[0095] Optional implementations of step 202 can be found in Figure 1 Optional implementations of step 102, and Figure 1 Other related parts in the embodiments involved will not be described in detail here.

[0096] In step 203, the depth feature information and the skeleton feature information in the first multimodal feature information are fused to obtain first multimodal fused feature information.

[0097] In some embodiments, the fusion operation may refer to a splicing operation, or other fusion operations, such as weighted fusion, etc., which are not limited in this disclosure and will not be described in detail. For example, the depth feature information and the skeleton feature information in the first multimodal feature information may be spliced ​​to obtain the first multimodal fusion feature information.

[0098] In step 204, a feature extractor in the video generation model is used to extract features from the first multimodal fusion feature information to obtain first feature information.

[0099] In a possible implementation, the feature extractor may be a spatial-temporal attention mechanism for feature extraction. In some embodiments, the feature extractor may include an attention block, and the attention block includes a spatial attention unit and a temporal attention unit. Exemplarily, the spatial attention unit is used to perform attention processing on the first multimodal fusion feature information to obtain the first intermediate feature information; the spatial dimension of the first intermediate feature information is transformed into the temporal dimension and then input into the temporal attention unit for attention processing to obtain the second intermediate feature information output by the temporal attention unit; the temporal dimension of the second intermediate feature information is transformed into the spatial dimension and then fused with the first intermediate feature information to obtain the first feature information.

[0100] In some embodiments, the number of attention blocks in the feature extractor can be multiple, that is: the feature extractor can include multiple attention blocks, the multiple attention blocks are connected in series, and each attention block includes a spatial attention unit and a temporal attention unit. The feature extractor can extract the first multimodal fusion feature information using a spatial-temporal attention mechanism to obtain the first feature information.

[0101] For example, taking the number of attention blocks as 4, Figure 3a As shown in the figure, the feature extractor includes 4 attention blocks. The feature extractor also includes a zero convolution (or zero convolution layer). The default output of the zero convolution can be 0. The output is gradually modified as the model is trained. The purpose is to not significantly modify the generation network in the video generation model at the beginning of the model. Figure 3aAs shown, each attention block includes a spatial attention unit and a temporal attention unit. The first multimodal fusion feature information c is input into the first attention block for attention processing. The output result of the first attention block is used as the input of the second attention block. The second attention block performs attention processing on the input. The output result of the second attention block is used as the input of the third attention block, and so on, until the output result of the last attention block is obtained, which is the first feature information obtained after the feature extractor extracts the first multimodal fusion feature information. Exemplary, as Figure 3b As shown, the spatial attention unit or the temporal attention unit may include a pooling layer (average pooling layer and maximum pooling layer) and a fully connected layer (or fully connected neural layer). More effective information can be obtained through the spatial-temporal attention mechanism.

[0102] In step 205, the depth feature information and the skeleton feature information in the second multimodal feature information are fused to obtain second multimodal fused feature information.

[0103] In some embodiments, the fusion operation may refer to a splicing operation, or other fusion operations, such as weighted fusion, etc., which are not limited in this disclosure and will not be described in detail. For example, the depth feature information and the bone feature information in the second multimodal feature information may be spliced ​​to obtain the second multimodal fusion feature information.

[0104] In step 206, a feature extractor is used to extract features from the second multimodal fusion feature information to obtain second feature information.

[0105] In some embodiments, the spatial attention unit of the feature extractor is used to perform attention processing on the second multimodal fusion feature information to obtain third intermediate feature information; the spatial dimension of the third intermediate feature information is transformed into a temporal dimension and then input into the temporal attention unit of the feature extractor for attention processing to obtain fourth intermediate feature information output by the temporal attention unit; the temporal dimension of the fourth intermediate feature information is transformed into a spatial dimension and then fused with the third intermediate feature information to obtain second feature information.

[0106] For example, Figure 3a Taking the feature extractor shown as an example, the second multimodal fusion feature information c is input into the first attention block for attention processing, the output result of the first attention block is used as the input of the second attention block, the second attention block performs attention processing on the input, and the output result of the second attention block is used as the input of the third attention block, and so on, until the output result of the last attention block is obtained, which is the second feature information obtained after the feature extractor performs feature extraction on the second multimodal fusion feature information.

[0107] It should be noted that, in some embodiments, the above steps 203 and 205 can be executed in an exchanged order or simultaneously, and the steps 204 and 206 can be executed in an exchanged order or simultaneously. In other words, the optional implementation method for obtaining the first characteristic information and the optional implementation method for obtaining the second characteristic information can be executed in an exchanged order or simultaneously.

[0108] In step 207, the first characteristic information and the second characteristic information are combined to form control condition characteristic information.

[0109] In some embodiments, the merging operation may refer to a splicing operation. Exemplarily, the first feature information and the second feature information may be spliced, and the features obtained after the splicing are used as the control condition feature information.

[0110] It should be noted that, since the driving video includes multiple frames of images and the reference image is a single frame of image, in order to ensure that the reference image has the same feature number as the driving video, after obtaining the first feature information of the reference image, the first feature information of the reference image can be copied according to the number of frames of the driving video, and the feature information obtained after the copying can be spliced ​​with the second feature information to obtain the control condition feature information. Exemplarily, when the number of frames of the driving video is 25, when obtaining the first feature information of the reference image, the first feature information of the reference image can be copied according to the number of frames of the driving video to obtain the first feature information of 25 frames of reference images, and the set of the first feature information of the 25 frames of reference images can be spliced ​​with the second feature information to obtain the control condition feature information.

[0111] In step 208, original noise is generated according to the reference image and the driving video.

[0112] Optional implementations of step 208 can be found in Figure 1 Optional implementations of step 104, and Figure 1 Other related parts in the embodiments involved will not be described in detail here.

[0113] In step 209, the reference image is encoded to obtain an embedding vector of the reference image.

[0114] Optional implementations of step 209 can be found in Figure 1 Optional implementations of step 105, and Figure 1 Other related parts in the embodiments involved will not be described in detail here.

[0115] It should be noted that, in some embodiments, the above-mentioned step of generating control condition characteristic information, step 208 and step 209 can be exchanged in order or performed simultaneously.

[0116] In step 210, noise prediction is performed based on the control condition feature information, the original noise, and the embedded vector of the reference image to obtain predicted noise that meets the control condition feature information.

[0117] In some embodiments, the video generation model includes a motion control module and a generation module, and the motion control module includes a conditional encoder. The encoder and the conditional encoder of the generation module can be used to encode the control condition feature information, the original noise, and the embedded vector of the reference image to obtain an encoding result, and the decoder of the generation module can be used to decode the encoding result to obtain a predicted noise that meets the control condition feature information.

[0118] In step 211, the parameters of the video generation model are adjusted according to the difference between the original noise and the predicted noise to obtain a trained video generation model.

[0119] Optional implementations of step 211 can be found in Figure 1 Optional implementations of step 107, and Figure 1 Other related parts in the embodiments involved will not be described in detail here.

[0120] In some embodiments, Figure 4 As shown, the above-mentioned optional implementation method of performing noise prediction based on the control condition feature information, the original noise and the embedded vector of the reference image to obtain the predicted noise that meets the control condition feature information may include the following steps:

[0121] In step 401, the control condition feature information and the original noise are fused to obtain fused feature information.

[0122] Exemplarily, the fusion operation may refer to an addition operation. Exemplarily, the control condition feature information and the feature information of the original noise may be added to obtain the fused feature information.

[0123] In step 402, a conditional encoder is used to encode the fused feature information and the embedded vector of the reference image to obtain third encoded feature information.

[0124] Exemplarily, the fused feature information is input into a conditional encoder, and the embedding vector and timestamp t of the reference image are input into the conditional encoder, and the conditional encoder is used to encode the fused feature information and the embedding vector of the reference image to obtain third encoded feature information.

[0125] In step 403, the encoder of the generation module is used to encode the feature information of the original noise and the embedding vector of the reference image to obtain fourth encoded feature information.

[0126] Exemplarily, the characteristic information of the original noise is input into the encoder of the generation module, the embedding vector and timestamp t of the reference image are input into the encoder of the generation module, and the encoder of the generation module is used to encode the characteristic information of the original noise and the embedding vector of the reference image to obtain fourth encoded characteristic information.

[0127] In step 404, the decoder of the generation module is used to decode the third encoded feature information, the fourth encoded feature information and the embedded vector of the reference image to obtain predicted noise that meets the control condition feature information.

[0128] Exemplarily, the third encoded feature information is added to the input of the corresponding layer of the decoder of the generation module through a zero convolution layer, the fourth encoded feature information, the embedding vector of the reference image and the timestamp t are input into the decoder of the generation module, and the decoder of the generation module is used to decode the third encoded feature information, the fourth encoded feature information and the embedding vector of the reference image to obtain the predicted noise.

[0129] In order to promote the potential feature interaction between the control conditions and the original noise and ensure the generation of stable, vivid and controllable videos, the present disclosure introduces a mutual information adapter. In some embodiments, the video generation model also includes multiple mutual information adapters, and the multiple mutual information adapters are respectively mapped to the number of layers of the encoder of the generation module and the number of layers of the conditional encoder, and each mutual information adapter includes multiple zero convolution layers. In an embodiment of the present disclosure, multiple mutual information adapters can be used to exchange information between each layer output of the encoder of the generation module and each layer output of the conditional encoder.

[0130] Exemplarily, each layer output of the encoder of the generation module and each layer output of the conditional encoder exchange information through a mutual information adapter, and each mutual information adapter is composed of four zero convolution modules. Figure 5 As shown, it is a structural example diagram of a mutual information adapter, and the mutual information adapter may include four zero convolution modules. Exemplarily, the mutual information adapter may include a first zero convolution, a second zero convolution, a third zero convolution, and a fourth zero convolution. For example, the first zero convolution and the third zero convolution may both be scale zero convolutions, and the second zero convolution and the fourth zero convolution may both be bias zero convolutions. Exemplarily, the first zero convolution is a first scale zero convolution, the second zero convolution is a first bias zero convolution, the third zero convolution is a second scale zero convolution, and the fourth zero convolution is a second bias zero convolution. In a possible implementation, the output x of the encoder corresponding layer of the generation module may be generated. z (i.e., original noise) is processed by the first Scale zero convolution and the first Bias zero convolution respectively, and the output x of the corresponding layer of the conditional encoder is converted to cThe conditional features are processed by the second Scale zero convolution and the second Bias zero convolution respectively, and the output result of the first Scale zero convolution is combined with the output x of the corresponding layer of the conditional encoder. c After the multiplication operation, the output result of the second Bias zero convolution is added to obtain the conditional feature information x′ processed by the mutual information adapter. c The output of the second scale zero convolution is combined with the output x of the corresponding layer of the encoder of the generation module z After the multiplication operation, the original noise feature information x′ processed by the mutual information adapter is obtained by adding the output result of the first Bias zero convolution. z Thus, the present disclosure introduces a mutual information adaptor to promote the potential feature interaction between the control condition and the original noise, thereby ensuring the generation of stable, vivid and controllable videos.

[0131] In order to facilitate those skilled in the art to more clearly understand the present disclosure, Figure 6 The training process of the video generation model is described in detail.

[0132] like Figure 6 As shown, the training process of the video generation model may include but is not limited to the following steps 601 to 605:

[0133] Step 601, randomly sample a 25-frame image sequence from the video as the driving video, and sample a random frame as the reference image. Use a feature extractor (structured as Figure 3a ) performs feature extraction on the depth map (such as the depth feature map) and the skeleton map (skeleton feature map) of the reference image to obtain the first feature information output by the feature extractor; using the feature extractor (structure as Figure 3a The feature extractor performs feature extraction on the depth map (such as the depth feature map) and the skeleton map (skeleton feature map) of the driving video to obtain the second feature information output by the feature extractor. The feature C obtained by copying the first feature information according to the number of frames of the driving video R and the second characteristic information C D After the merging operation, it is used as the control condition feature information (also called conditional noise).

[0134] Step 602: After extracting features from the driving video using the VAE encoder, Gaussian noise is added according to the timestamp t to generate the second encoded feature information z after adding noise. t The reference image is used to extract the reference feature ε(I) using the VAE encoder, and the reference feature ε(I) is copied according to the number of frames of the driving video and then compared with z t Merge to generate the original noise z′ t The conditional noise (i.e., the control condition feature information) is processed in the same way and is compared with the original noise z′t Add and generate fused feature information c.

[0135] Step 603, use the CLIP network to extract image features from the reference image, obtain the embedding vector C(I) of the reference image, and input the embedding vector C(I) and the timestamp t into each attention layer of the generation module, and input the embedding vector C(I) and the timestamp t into each attention layer of the conditional encoder.

[0136] Step 604: convert the original noise z′ t Input to the generation module (including encoder E Gen and decoder D Gen ), the fused feature information c is input into the conditional encoder E Ctrl and the conditional encoder E Ctrl The output of is added to the generator module decoder D through a zero convolution layer. Gen The encoder E of the generation module Gen Each layer output is combined with the conditional encoder E Ctrl There is a mutual information adapter between each layer output, through which the encoder E Gen The output of the corresponding layer and the conditional encoder E Ctrl The output of the corresponding layer exchanges information. Gen Output predicted noise. The original noise z′ is calculated by predicting the noise t Denoising is performed to obtain a predicted video. The default output of the zero convolution layer can be zero, and the output of the zero convolution layer is gradually modified as the network is trained, in order to not significantly modify the parameters of the generation module at the beginning of the network.

[0137] Step 605, according to the difference between the predicted noise and the above-mentioned Gaussian noise, the parameters of the video generation model are adjusted to obtain a trained video generation model. Exemplarily, the inter-frame difference of the original noise can be obtained, and the inter-frame difference of the predicted noise can be obtained; the inter-frame variance loss value is generated according to the inter-frame difference of the original noise and the inter-frame difference of the predicted noise; the parameters of the video generation model are adjusted according to the inter-frame variance loss value, for example, the inter-frame variance loss value is used as the model loss value, and the parameters of the video generation model are updated using the model loss value. Exemplarily, the corresponding gradient information can be calculated according to the inter-frame variance loss value using a gradient descent algorithm, and the parameters of the video generation model can be updated according to the gradient information. For example, the value of each parameter to be adjusted in the video generation model can be updated according to the gradient information, so that it moves in the opposite direction of the gradient, thereby reducing the model loss value; wherein the gradient information can represent the direction in which the loss function changes fastest. It can be understood that other parameter adjustment methods can also be used to adjust the parameters of the video generation model, which is not limited in this disclosure and will not be repeated.

[0138] It is worth noting that the training process of the video generation model involved in the embodiment of the present disclosure adopts an end-to-end strategy. The video generation module includes a motion control module and a generation module. The video generation module may also include a mutual information adapter. The motion control module and the mutual information adapter are used to achieve the unification of motion control and appearance. By fusing the features of the control condition latent variables and the video noise latent variables, the video generator based on the diffusion model is effectively integrated with the motion control module. In addition, the embodiment of the present disclosure takes the human skeleton and depth features as control conditions, and proposes a multimodal fusion module to simultaneously extract depth and skeleton features through a space-time attention mechanism. In addition, the embodiment of the present disclosure uses inter-frame variance loss to constrain the consistency and continuity between the input video features and the latent features generated by the video generator, so as to further enhance the temporal stability.

[0139] The disclosed embodiment also proposes a video generation method. Figure 7 Schematic diagram of a video generation method provided by an embodiment of the present disclosure. Figure 7 As shown, the video generation method may include but is not limited to the following steps.

[0140] In step 701, an image to be processed is obtained, and a reference action feature sequence is obtained.

[0141] Exemplarily, the image to be processed may be an image that needs to be driven. The reference action feature sequence may be a given action feature sequence.

[0142] In step 702, a pre-trained video generation model is used to perform noise prediction on the image to be processed and the action feature sequence to obtain predicted noise.

[0143] In an embodiment of the present disclosure, the image to be processed and the action feature sequence may be input into a video generation model for noise prediction to obtain predicted noise.

[0144] Exemplarily, the video generation model can be pre-trained, for example, it can be trained using the video generation model training method described in any of the aforementioned embodiments. The optional implementation methods of the video generation model training process can refer to the implementable methods of the training method described in any of the aforementioned embodiments, which will not be repeated here.

[0145] In step 703, a denoising module based on the video generation model performs multiple rounds of iterative denoising processing using the predicted noise to obtain video generation feature information.

[0146] In some embodiments, the denoising module of the video generation model can be constructed using the denoising diffusion implicit model framework DDIM, which can improve the efficiency of reasoning. Exemplarily, after obtaining the predicted noise, the predicted noise can be input into the denoising module of the video generation model, and the final video generation feature information can be obtained after multiple rounds of iterative denoising processing using the predicted noise.

[0147] In step 704, the video generation feature information is decoded using the VAE decoder of the video generation model to obtain a video.

[0148] Exemplarily, the video generation feature information is input into the VAE decoder for decoding to obtain the video.

[0149] By implementing the embodiments of the present disclosure, the trained video generation model is directly used to generate videos, thereby achieving end-to-end human motion video generation. When training the video generation model, by using the depth map and the skeleton map as control conditions, it is possible to achieve the result of generating human body videos according to multimodal condition control information, and the controllability and smoothness are greatly enhanced, thereby better ensuring the stability of the generated movements. In addition, the embodiments of the present disclosure use multimodal motion guidance to accurately capture the appearance and motion details, and apply classifier-free guidance to ensure that the generated results are consistent with the latent space specified by the control conditions, thereby enabling stable action video generation even when the control conditions are jittery.

[0150] Figure 8 This is a block diagram of a video generation model training device provided by an embodiment of the present disclosure. Figure 8 As shown, the video generation model training device may include: a first feature extraction module 810, a second feature extraction module 820, a first generation module 830, a second generation module 840, an encoding module 850, a noise prediction module 860 and a parameter adjustment module 870.

[0151] The first feature extraction module 810 is used to extract features from the reference image to obtain first multimodal feature information of the reference image, where the first multimodal feature information includes depth feature information and bone feature information.

[0152] The second feature extraction module 820 is used to extract features from the driving video to obtain second multimodal feature information, where the second multimodal feature information includes depth feature information and bone feature information.

[0153] The first generating module 830 is used to generate control condition feature information according to the first multimodal feature information and the second multimodal feature information.

[0154] The second generation module 840 is used to generate original noise according to the reference image and the driving video. In some embodiments, the video generation model includes a variational autoencoder VAE; the second generation module 840 is used to: use the VAE encoder to encode the reference image to obtain first encoding feature information of the reference image; use the VAE encoder to encode the driving video to obtain second encoding feature information of the driving video; add noise to the second encoding feature information and merge it with the first encoding feature information to obtain original noise.

[0155] The encoding module 850 is used to perform encoding processing on the reference image to obtain an embedding vector of the reference image.

[0156] The noise prediction module 860 is used to perform noise prediction based on the control condition feature information, the original noise and the embedded vector of the reference image to obtain the predicted noise that meets the control condition feature information.

[0157] In some embodiments, the video generation model includes a motion control module and a generation module, the motion control module includes a conditional encoder; the noise prediction module 860 is used to: fuse the control condition feature information and the feature information of the original noise to obtain fused feature information; use the conditional encoder to encode the fused feature information and the embedded vector of the reference image to obtain third encoded feature information; use the encoder of the generation module to encode the feature information of the original noise and the embedded vector of the reference image to obtain fourth encoded feature information; use the decoder of the generation module to decode the third encoded feature information, the fourth encoded feature information and the embedded vector of the reference image to obtain predicted noise that meets the control condition feature information.

[0158] The parameter adjustment module 870 is used to adjust the parameters of the video generation model according to the difference between the original noise and the predicted noise to obtain the trained video generation model. In some embodiments, the parameter adjustment module 870 is used to: obtain the inter-frame difference of the original noise and obtain the inter-frame difference of the predicted noise; generate an inter-frame variance loss value according to the inter-frame difference of the original noise and the inter-frame difference of the predicted noise; and adjust the parameters of the video generation model according to the inter-frame variance loss value.

[0159] In some embodiments, the video generation model includes a feature extractor; Fig. 9 As shown, the first generation module 930 includes: a first fusion unit 931, a first feature extraction unit 932, a second fusion unit 933, a second feature extraction unit 934 and a merging unit 935. Among them, the first fusion unit 931 is used to fuse the depth feature information and the bone feature information in the first multimodal feature information to obtain the first multimodal fusion feature information; the first feature extraction unit 932 is used to use the feature extractor to extract the first feature information from the first multimodal fusion feature information to obtain the first feature information; the second fusion unit 933 is used to fuse the depth feature information and the bone feature information in the second multimodal feature information to obtain the second multimodal fusion feature information; the second feature extraction unit 934 is used to use the feature extractor to extract the second multimodal fusion feature information to obtain the second feature information; the merging unit 935 is used to merge the first feature information and the second feature information as the control condition feature information. Among them, Fig. 9 Zhong 910~970 and Figure 8 810 to 870 have the same function and structure.

[0160] In some embodiments, the feature extractor includes an attention block, which includes a spatial attention unit and a temporal attention unit; the first feature extraction unit 932 is used to: use the spatial attention unit to perform attention processing on the first multimodal fusion feature information to obtain first intermediate feature information; transform the spatial dimension of the first intermediate feature information into a temporal dimension and input it into the temporal attention unit for attention processing to obtain second intermediate feature information output by the temporal attention unit; transform the temporal dimension of the second intermediate feature information into a spatial dimension and perform a fusion operation with the first intermediate feature information to obtain first feature information.

[0161] In some embodiments, the second feature extraction unit 934 is used to: use the spatial attention unit to perform attention processing on the second multimodal fusion feature information to obtain third intermediate feature information; transform the spatial dimension of the third intermediate feature information into a temporal dimension and input it into the temporal attention unit for attention processing to obtain fourth intermediate feature information output by the temporal attention unit; transform the temporal dimension of the fourth intermediate feature information into a spatial dimension and perform a fusion operation with the third intermediate feature information to obtain second feature information.

[0162] In some embodiments, the video generation model further includes a plurality of mutual information adapters, and the plurality of mutual information adapters are respectively mapped to the number of layers of the encoder of the generation module and the number of layers of the conditional encoder. Fig.10 As shown, the video generation model training device also includes: an information exchange module 1080. The information exchange module 1080 is used to use multiple mutual information adapters to exchange information between each layer output of the encoder of the generation module and each layer output of the conditional encoder through a mutual information adapter. Fig.10 Zhong 1010~1070 and Fig. 9 910 to 970 have the same function and structure.

[0163] In some embodiments, each mutual information adapter includes multiple zero convolution layers, and the multiple zero convolution layers include a first zero convolution, a second zero convolution, a third zero convolution, and a fourth zero convolution; wherein, the information exchange module 1080 exchanges information between each layer output of the encoder of the generation module and each layer output of the conditional encoder through a mutual information adapter. A possible implementation method includes: processing the output of the corresponding layer of the encoder of the generation module through the first zero convolution and the second zero convolution respectively; processing the output of the corresponding layer of the conditional encoder through the third zero convolution and the fourth zero convolution respectively; multiplying the output result of the first zero convolution with the output of the corresponding layer of the conditional encoder, and adding the output result of the fourth zero convolution to obtain the conditional feature information processed by the corresponding mutual information adapter; multiplying the output result of the third zero convolution with the output of the corresponding layer of the encoder of the generation module, and adding the output result of the second zero convolution to obtain the original noise feature information processed by the corresponding mutual information adapter.

[0164] In some embodiments, the first zero convolution and the third zero convolution are both Scale zero convolutions, and the second zero convolution and the fourth zero convolution are both Bias zero convolutions.

[0165] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0166] Fig.11FIG. 1 is a block diagram of a video generation device provided by an embodiment of the present disclosure. Fig.11 As shown, the video generating device may include: an acquisition module 1101 , a noise prediction module 1102 , a denoising processing module 1103 , and a decoding processing module 1104 .

[0167] The acquisition module 1101 is used to acquire the image to be processed and obtain the reference action feature sequence

[0168] The noise prediction module 1102 is used to perform noise prediction on the image to be processed and the action feature sequence using a pre-trained video generation model to obtain predicted noise. The video generation model is a model trained using the training method described in any of the above embodiments.

[0169] The denoising processing module 1103 is a denoising module based on a video generation model, which performs multiple rounds of iterative denoising processing using predicted noise to obtain video generation feature information.

[0170] The decoding processing module 1104 is used to use the VAE decoder of the video generation model to decode the video generation feature information to obtain the video.

[0171] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0172] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0173] like Fig.12 , is a block diagram of an electronic device provided by an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0174] like Fig.12As shown, the electronic device includes: one or more processors 1201, memory 1202, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.12 A processor 1201 is taken as an example.

[0175] The memory 1202 is a non-transient computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the video generation module training method and / or video generation method provided by the present disclosure. The non-transient computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to perform the video generation module training method and / or video generation method provided by the present disclosure.

[0176] The memory 1202 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the video generation module training method and / or the program instructions / modules corresponding to the video generation method in the embodiment of the present disclosure (for example, the attached Figure 8 The first feature extraction module 810, the second feature extraction module 820, the first generation module 830, the second generation module 840, the encoding module 850, the noise prediction module 860 and the parameter adjustment module 870 are shown in FIG. Fig.11 The processor 1201 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1202, that is, the video generation module training method and / or the video generation method in the above method embodiment are implemented.

[0177] The memory 1202 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1202 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1202 may optionally include a memory remotely arranged relative to the processor 1201, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The electronic device may further include: an input device 1203 and an output device 1204. The processor 1201, the memory 1202, the input device 1203 and the output device 1204 may be connected via a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0179] The input device 1203 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1204 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0180] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0183] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0184] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0185] It should be noted that any process or method description in the flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations, in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0186] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0187] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0188] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0189] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0190] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A video generation model training method, It is characterized in that include: Performing feature extraction on a reference image to obtain first multimodal feature information of the reference image, wherein the first multimodal feature information includes depth feature information and bone feature information; Extracting features from the driving video to obtain second multimodal feature information, where the second multimodal feature information includes depth feature information and skeleton feature information; generating control condition feature information according to the first multimodal feature information and the second multimodal feature information; generating original noise according to the reference image and the driving video; Performing encoding processing on the reference image to obtain an embedding vector of the reference image; Perform noise prediction according to the control condition characteristic information, the original noise and the embedded vector of the reference image to obtain predicted noise that meets the control condition characteristic information; According to the difference between the original noise and the predicted noise, the parameters of the video generation model are adjusted to obtain a trained video generation model.

2. The method according to claim 1, It is characterized in that The video generation model includes a feature extractor; the generating control condition feature information according to the first multimodal feature information and the second multimodal feature information includes: Fusing the depth feature information and the skeleton feature information in the first multimodal feature information to obtain first multimodal fused feature information; Using the feature extractor to extract features from the first multimodal fusion feature information to obtain first feature information; fusing the depth feature information and the skeleton feature information in the second multimodal feature information to obtain second multimodal fused feature information; Using the feature extractor to extract features from the second multimodal fusion feature information to obtain second feature information; The first characteristic information and the second characteristic information are combined to form the control condition characteristic information.

3. The method according to claim 2, It is characterized in that The feature extractor includes an attention block, wherein the attention block includes a spatial attention unit and a temporal attention unit; The step of extracting features from the first multimodal fusion feature information using the feature extractor to obtain first feature information includes: Using the spatial attention unit to perform attention processing on the first multimodal fusion feature information to obtain first intermediate feature information; Transforming the spatial dimension of the first intermediate feature information into a temporal dimension and inputting the result into the temporal attention unit for attention processing to obtain second intermediate feature information output by the temporal attention unit; Transforming the time dimension of the second intermediate feature information into a spatial dimension and then fusing it with the first intermediate feature information to obtain the first feature information; The step of extracting features from the second multimodal fusion feature information using the feature extractor to obtain second feature information includes: Using the spatial attention unit to perform attention processing on the second multimodal fusion feature information to obtain third intermediate feature information; Transforming the spatial dimension of the third intermediate feature information into a temporal dimension and inputting the resultant into the temporal attention unit for attention processing, thereby obtaining fourth intermediate feature information output by the temporal attention unit; The time dimension of the fourth intermediate feature information is transformed into a space dimension and then fused with the third intermediate feature information to obtain the second feature information.

4. The method according to claim 1, It is characterized in that The video generation model includes a variational autoencoder VAE; the generating original noise according to the reference image and the driving video includes: Using the VAE encoder to encode the reference image to obtain first encoding feature information of the reference image; Using the VAE encoder to encode the driving video to obtain second encoding feature information of the driving video; The second coding feature information is added with noise and then combined with the first coding feature information to obtain the original noise.

5. The method according to claim 1, It is characterized in that The video generation model includes a motion control module and a generation module, and the motion control module includes a conditional encoder; the noise prediction is performed according to the control condition feature information, the original noise and the embedded vector of the reference image to obtain the predicted noise that meets the control condition feature information, including: Fusing the control condition feature information with the feature information of the original noise to obtain fused feature information; Using the conditional encoder to encode the fused feature information and the embedding vector of the reference image to obtain third encoded feature information; Using the encoder of the generation module to encode the feature information of the original noise and the embedding vector of the reference image to obtain fourth encoded feature information; The decoder of the generation module is used to decode the third encoded feature information, the fourth encoded feature information and the embedded vector of the reference image to obtain the predicted noise of the feature information that meets the control condition.

6. The method according to claim 5, It is characterized in that The video generation model further includes a plurality of mutual information adapters, and the plurality of mutual information adapters are respectively mapped to the number of layers of the encoder of the generation module and the number of layers of the conditional encoder; the method further includes: Each layer output of the encoder of the generation module and each layer output of the conditional encoder are exchanged with each other through a mutual information adapter.

7. The method according to claim 6, It is characterized in that Each of the mutual information adapters includes a plurality of zero convolution layers, wherein the plurality of zero convolution layers include a first zero convolution, a second zero convolution, a third zero convolution, and a fourth zero convolution; wherein the information exchange between each layer output of the encoder of the generation module and each layer output of the conditional encoder through one of the mutual information adapters includes: Processing the output of the encoder corresponding layer of the generation module through the first zero convolution and the second zero convolution respectively; Processing the output of the corresponding layer of the conditional encoder through the third zero convolution and the fourth zero convolution respectively; After multiplying the output result of the first zero convolution with the output of the corresponding layer of the conditional encoder, and adding the output result of the fourth zero convolution to obtain conditional feature information processed by the corresponding mutual information adapter; After multiplying the output result of the third zero convolution with the output of the corresponding layer of the encoder of the generation module, and adding the output result of the second zero convolution, the original noise feature information processed by the corresponding mutual information adapter is obtained.

8. The method according to claim 7, It is characterized in that The first zero convolution and the third zero convolution are both Scale zero convolutions, and the second zero convolution and the fourth zero convolution are both Bias zero convolutions.

9. The method according to any one of claims 1 to 8, It is characterized in that The adjusting the parameters of the video generation model according to the difference between the original noise and the predicted noise includes: Obtaining an inter-frame difference of the original noise, and obtaining an inter-frame difference of the predicted noise; Generating an inter-frame variance loss value according to the inter-frame difference value of the original noise and the inter-frame difference value of the predicted noise; The parameters of the video generation model are adjusted according to the inter-frame variance loss value.

10. A video generation method, It is characterized in that include: Obtain an image to be processed and obtain a reference action feature sequence; Using a pre-trained video generation model, noise prediction is performed on the image to be processed and the action feature sequence to obtain predicted noise; Based on the denoising module of the video generation model, multiple rounds of iterative denoising processing are performed using the predicted noise to obtain video generation feature information; Decoding the video generation feature information using a VAE decoder of the video generation model to obtain a video; The video generation model is a model trained by the training method described in any one of claims 1 to 9.

11. A video generation model training device, It is characterized in that include: A first feature extraction module, configured to extract features from a reference image to obtain first multimodal feature information of the reference image, wherein the first multimodal feature information includes depth feature information and bone feature information; A second feature extraction module, used to extract features from the driving video to obtain second multimodal feature information, where the second multimodal feature information includes depth feature information and bone feature information; A first generating module, configured to generate control condition feature information according to the first multimodal feature information and the second multimodal feature information; A second generating module, used for generating original noise according to the reference image and the driving video; An encoding module, used for encoding the reference image to obtain an embedding vector of the reference image; A noise prediction module, used for performing noise prediction according to the control condition characteristic information, the original noise and the embedded vector of the reference image, to obtain predicted noise that meets the control condition characteristic information; The parameter adjustment module is used to adjust the parameters of the video generation model according to the difference between the original noise and the predicted noise to obtain a trained video generation model.

12. A video generating device, It is characterized in that include: An acquisition module is used to acquire an image to be processed and obtain a reference action feature sequence; A noise prediction module, used for performing noise prediction on the image to be processed and the action feature sequence using a pre-trained video generation model to obtain predicted noise; A denoising processing module, which is used to perform multiple rounds of iterative denoising processing based on the denoising module of the video generation model using the predicted noise to obtain video generation feature information; A decoding processing module, used to decode the video generation feature information using the VAE decoder of the video generation model to obtain a video; The video generation model is a model trained by the training method described in any one of claims 1 to 9.

13. An electronic device, It is characterized in that include: at least one processor; a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9 and 10.

14. A storage medium, It is characterized in that The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 9 and 10.

15. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 and 10 are implemented.

Images