Video generation method and apparatus, electronic device, medium, and program product

Abstract

The application discloses a video generation method and device, electronic equipment, medium and program product, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring reference data; acquiring multi-modal features according to the reference data; the multi-modal reference features comprise image features, text features and posture features; and generating a target video corresponding to the reference data according to the multi-modal features. The target video is generated according to the multi-modal features comprising image features, text features and posture features, multi-conditional driving of the target video generation is realized, the obtained multi-modal features can reflect more real video content, the generated video content is more realistic, has better feature preservation effect, and sudden change frames are avoided, and the video quality is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically to a video generation method, apparatus, electronic device, medium, and program product. Background Technology

[0002] With the development of image processing technology in recent years, there is a wide demand for AI-based digital humans in applications such as remote video conferencing, virtual character generation, and animated video creation. How to construct realistic and high-quality virtual characters has become a widely discussed issue.

[0003] Existing face video generation technologies, such as face video generation technology based on dual U-Net architecture, propose to extract corresponding facial key points from the input audio signal and generate conversational face videos through the interaction between the two U-Nets. Most other existing video generation methods are driven by single conditions such as images or text descriptions, resulting in poor video content quality and easy distortion. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, electronic device, medium, and program product for generating face videos, in order to overcome the shortcomings of the prior art in generating high-quality, realistic videos.

[0005] To achieve the above objectives, the first aspect of this application provides a video generation method, the video generation method comprising:

[0006] Obtain reference data;

[0007] Based on the reference data, multimodal features are obtained; the multimodal reference features include: image features, text features, and pose features;

[0008] Based on the multimodal features, a target video corresponding to the reference data is generated.

[0009] Based on the aforementioned technical means, target videos are generated using multimodal features, including image features, text features, and pose features. The multi-condition driving of target video generation allows the extracted multimodal features to reflect more realistic video content, making the generated video content more lifelike and having better feature preservation effects, thereby improving video quality.

[0010] In some feasible embodiments, obtaining multimodal features based on the reference data includes:

[0011] Based on the reference data and auxiliary data, image features, text features, and pose features corresponding to the reference data are obtained; the reference data includes one or more of the following: reference image, key point sequence, and text prompt information; the auxiliary data includes one or more of the following: image auxiliary data, text auxiliary data, and key point auxiliary data.

[0012] Based on the above technical means, when the reference data only includes one or two of the following: reference image, key point sequence, and text prompt information, the corresponding auxiliary data is used to replace the unprovided reference data in order to obtain the corresponding data features, ensuring the normal progress of multi-condition driven video generation. At the same time, the auxiliary data does not affect the video generation result, and the generated video has a high degree of consistency with the input reference data.

[0013] In some feasible embodiments, the image features are generated based on a DIT diffusion network; the method for acquiring the image features includes:

[0014] The reference image or image auxiliary data is three-dimensionally compressed based on the first variational autoencoder to obtain the variational probability distribution of the reference image or the image auxiliary data.

[0015] The variational probability distribution is converted into a series of labels;

[0016] Text features are used as key and value inputs for each feature extraction spatial attention block, and the output of the previous feature extraction spatial attention block is used as the query input for the next feature extraction spatial attention block. The spatial attention features of different feature extraction spatial attention blocks are calculated progressively, and the spatial attention features of multiple feature extraction spatial attention blocks are used as image features; wherein, the query input of the first feature extraction spatial attention block is the series of flags.

[0017] Based on the above technical means, the face image is compressed including the time dimension so that the obtained image features can fully consider the time dynamics. A series of tags are used to drive the spatial attention calculation of text features so that the obtained image features are more consistent with the text features. By performing spatial attention calculation on the variational probability distribution of the reference image after time dimension compression, the obtained image features have both time dimension information and spatial dimension information, which improves the feature accuracy and coherence of the generated video.

[0018] In some feasible embodiments, the method for obtaining the pose features includes:

[0019] Based on the second variational autoencoder, the key point sequence or key point auxiliary data is three-dimensionally compressed to obtain a first sequence with the same size as the noise vector.

[0020] Based on the pose adaptation module, feature extraction is performed on the first sequence to obtain a first feature with the same size as the noise vector;

[0021] The first feature is added element-wise to the noise vector to obtain the pose feature.

[0022] Based on the above technical means, the corresponding pose features in the video can be extracted through the key point sequence. The pose features are used to drive the generated video frames, ensuring that the content in the generated video can perform the corresponding actions. The key point sequence is compressed, including the time dimension, so that the obtained pose features can fully take into account the temporal dynamics and enhance the coherence of the generated video.

[0023] In some feasible embodiments, generating the target video corresponding to the reference data based on the multimodal features includes:

[0024] The spatiotemporal information of the multimodal features is calculated using a backbone network; the backbone network is constructed based on a DIT diffusion network.

[0025] The spatiotemporal information is normalized to obtain variance prediction;

[0026] The variance prediction is decoded in pixel space to obtain the target video.

[0027] Based on the above technical means, by using a DIT-based diffusion network to calculate the spatiotemporal information of multimodal features, information with more coherent temporal relationships and spatial relationships that are more in line with natural laws can be extracted from multimodal features. After normalization processing, variance prediction is obtained. The interaction between frames in the target video generated based on variance prediction is more accurate, and the spatial relationship between the content in each frame of the video is more reasonable, thus improving the video quality.

[0028] In some feasible embodiments, the step of using a backbone network to calculate the spatiotemporal information of the multimodal features includes:

[0029] The text features are used as the key and value inputs for each denoised spatial attention block and each denoised temporal attention block. The output of the previous denoised spatial attention block is used as the query input for the next denoised temporal attention block. The output of the previous denoised temporal attention block and the image features are added element-wise as the query input for the next denoised spatial attention block. The first denoised spatial attention block is located at the beginning of the backbone network, and the query input for the first denoised spatial attention block is the output of the pose features and the image features added element-wise.

[0030] Based on the aforementioned technical methods, videos are generated using multiple reference data, supporting multi-condition guided face video generation. Image features are extracted using a diffusion network, and video generation is also achieved using a diffusion network, realizing video generation based on a dual diffusion network. The denoising temporal attention block can extract temporal relationship information from multimodal features, while the denoising spatial attention block can extract information whose spatial relationships more conform to natural laws. This information can accurately reflect the interaction between frames in subsequent videos and the spatial relationship between the content in each frame, thereby improving the quality of the generated video and avoiding frames with abrupt changes in key features.

[0031] A second aspect of this application provides a video generation apparatus, the video generation apparatus comprising:

[0032] The data acquisition unit is used to acquire reference data;

[0033] The feature extraction unit is used to obtain multimodal features based on the reference data; the multimodal reference features include: image features, text features, and pose features;

[0034] The video generation unit is used to generate a target video corresponding to the reference data based on the multimodal features.

[0035] Based on the aforementioned technical means, the device generates target videos based on multimodal features including image features, text features, and pose features. The multi-condition driving of target video generation and the extracted multimodal features can reflect more realistic video content, making the generated video content more lifelike and having better feature preservation effect, thereby improving video quality.

[0036] A third aspect of this application provides an electronic device, comprising:

[0037] The memory is configured to store instructions; and

[0038] The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the video generation method.

[0039] A fourth aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform the video generation method.

[0040] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the video generation method.

[0041] The above technical solution provides a video generation method that generates target videos based on multimodal features including image features, text features, and pose features. The multi-condition driven target video generation and the extracted multimodal features can reflect more realistic video content, making the generated video content more lifelike and having better feature preservation effect, thereby improving video quality.

[0042] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0043] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0044] Figure 1 This illustration schematically shows a video generation method flowchart according to an embodiment of this application;

[0045] Figure 2 This illustration schematically shows an overall route map for video generation provided according to one embodiment of this application.

[0046] Figure 3 This illustration schematically shows a process diagram of the spatial transformation module extracting spatial attention features according to an embodiment of this application;

[0047] Figure 4 This illustration schematically shows a process diagram of the spatiotemporal information extraction by the spatiotemporal conversion module according to an embodiment of this application.

[0048] Figure 5 The illustration shows a schematic diagram of the structure of a video generation apparatus according to an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0050] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0051] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0052] Figure 1 The illustration shows a schematic flowchart of a video generation method according to an embodiment of this application. This method can be applied to terminal devices such as in-vehicle systems, smartphones, PDAs, tablets, laptops, all-in-one computers, and autonomous driving devices. It is understood that the dialogue method provided in this disclosure can also be applied to other scenarios.

[0053] like Figure 1 As shown in the figure, this application provides a video generation method, which may include the following steps.

[0054] S1: Obtain reference data. In some feasible embodiments, the reference data serves as the basis for video generation. The reference data needs to be related to the video to be generated. For example, to generate a face video, the reference data needs to provide information such as the face shape, eye shape, hairstyle, and hair color; to generate an animal video, the reference data needs to provide information such as the animal's species and fur color. The reference data includes one or more of the following: reference images, key point sequences, and text prompts. The reference image is the image on which the video generation is based. For example, to generate a face video, the reference image can be an image containing a face, which may include the face shape, eye shape, hairstyle, and hair color, and can be an ID photo, headshot, selfie, etc. To generate a dog-related video, the reference image can be an image including a dog's head, which may include the dog's fur color, mouth, eyes, ears, etc.

[0055] A keypoint sequence refers to a sequence of changes in key positions or feature points identified and recorded during a series of actions or events. These keypoints can be human joints, facial feature points, or specific locations of other objects. Keypoint sequences have wide applications in various fields, particularly in human motion recognition, pose estimation, and expression recognition. In this embodiment, the keypoint sequence can be an animal motion keypoint sequence, a facial keypoint sequence, a human joint keypoint sequence, etc. Specifically, a facial keypoint sequence refers to a series of image frames in which facial keypoints have been located. Facial keypoints are important facial feature points located in a face image; these feature points typically include the positions of the eyes, eyebrows, nose, mouth, and facial contours. Facial keypoints are fundamental to face-related problems, helping models understand facial expressions, poses, and other information, thereby enabling more intelligent face recognition and human-computer interaction functions. If a facial keypoint sequence is used for face video generation, it can be used to control the actions, expressions, and shapes of the generated face.

[0056] Text prompts are text descriptions of the content to be generated in the video. For example, when generating a face video based on text prompts, the text prompts could be descriptions of the face shape, eye shape, hair color, etc. Similarly, when generating a landscape video based on text prompts, the text prompts could be descriptions of the time, weather, and included scenery.

[0057] S2: Obtain multimodal features based on the reference data; the multimodal reference features include: image features, text features, and pose features. In some feasible embodiments, multimodal features refer to features extracted from different reference data. Specifically, image features are features extracted from reference images, text features are features extracted from text prompts, and pose features are features extracted from keypoint sequences superimposed with a noise vector.

[0058] In some feasible embodiments, obtaining multimodal features based on the reference data includes:

[0059] Based on the reference data and auxiliary data, image features, text features, and pose features corresponding to the reference data are obtained; the reference data includes one or more of the following: reference image, key point sequence, and text prompt information; the auxiliary data includes one or more of the following: image auxiliary data, text auxiliary data, and key point auxiliary data.

[0060] To ensure the acquisition of multimodal reference features during video generation, auxiliary data is used to guarantee the integrity of these features. The auxiliary data consists entirely of zeros. After feature extraction, the auxiliary data yields corresponding zero features. Video generation is based on these zero features, ensuring the quality of the video generated from the existing reference data is not affected. In this embodiment, if the acquired reference data only includes one or two of the following: a reference image, a keypoint sequence, and text prompts, the corresponding auxiliary data is used to replace the unprovided reference data to obtain the corresponding data features. For example, if the reference data only includes text prompts, then text prompts, image auxiliary data, and keypoint auxiliary data are used to extract multimodal reference features; if the reference data includes keypoint sequences and a reference image, then text auxiliary data, keypoint sequences, and a reference image are used to extract multimodal reference features, and so on. The text auxiliary data, keypoint auxiliary data, and image auxiliary data are pre-configured data. The text auxiliary data has the same shape as the text prompts, the keypoint auxiliary data has the same shape as the keypoint sequences, and the image auxiliary data has the same shape as the reference image.

[0061] Based on the above technical means, when the reference data only includes one or two of the following: reference image, key point sequence, and text prompt information, the corresponding auxiliary data is used to replace the unprovided reference data in order to obtain the corresponding data features, ensuring the normal progress of multi-condition driven video generation. At the same time, the auxiliary data does not affect the video generation result, and the generated video has a high degree of consistency with the input reference data.

[0062] In some feasible embodiments, the image features are generated based on a Diffusion Transformer (DiT) network. Diffusion models are generative models based on a diffusion process. They learn the distribution characteristics of data by simulating the transition of data from an ordered state to a disordered state and can reverse this process to generate new data samples. That is, by simulating the physical diffusion process, noise is gradually added to the data, and then the reverse process is learned to gradually recover the original data from the noise, thereby achieving high-quality generation results. Diffusion models have received widespread attention and application in recent years, especially showing great potential in image synthesis, style transfer, and data augmentation. In the embodiments of this application, the diffusion model used is the DiT (Diffusion Transformer) model. DiT is a class of diffusion models based on the Transformer architecture, which replaces the U-Net typically used in diffusion models as the backbone network with a Transformer. DiT studies variants of the Transformer that handle conditional inputs in different ways, such as through adaptive layer normalization, cross-attention, and additional input tokens.

[0063] The method for obtaining the image features includes:

[0064] 1) The reference image or image auxiliary data is three-dimensionally compressed based on the first variational autoencoder to obtain the variational probability distribution of the reference image or the image auxiliary data. In some feasible embodiments, a pre-trained slice variational autoencoder (Slice VAE) is used to perform three-dimensional compression on the face reference image. In this embodiment, three dimensions refer to the length, width, and time dimensions. The slice variational autoencoder segments the face reference image in the time dimension to obtain multiple independent parts. Variational inference is performed on each part to obtain the variational probability distribution of each part.

[0065] 2) The variational probability distribution is converted into a series of tokens. In this embodiment, an embedding method is used to convert the discrete variational probability distribution into a continuous high-order vector of tokens.

[0066] 3) Text features are used as the key input (K) and value input (V) of each feature extraction spatial attention block, and the output of the previous feature extraction spatial attention block is used as the query input (Q) of the next feature extraction spatial attention block. The spatial attention features of different feature extraction spatial attention blocks are calculated progressively, and the spatial attention features of multiple feature extraction spatial attention blocks are used as image features. The query input of the first feature extraction spatial attention block is the series of flags. In spatial attention calculation, the query input represents the feature at each position in the output feature map, used for matching with the key input. The key input represents the feature at each position in the input feature map, and the value input contains specific information related to the key input, used to generate the output feature map. Spatial attention calculation based on text features and the variational probability distribution of the reference image can extract more accurate content from the reference image. The spatial attention features of spatial attention blocks at different depths characterize the importance of content features in the image; the deeper the block, the more important the content features.

[0067] The extracted image features are stored in a list (bank) for subsequent video generation.

[0068] Based on the above technical means, the face image is compressed including the time dimension so that the obtained image features can fully consider the time dynamics. A series of tags are used to drive the spatial attention calculation of text features so that the obtained image features are more consistent with the text features. By performing spatial attention calculation on the variational probability distribution of the reference image after time dimension compression, the obtained image features have both time dimension information and spatial dimension information, which improves the feature accuracy and coherence of the generated video.

[0069] In some feasible embodiments, the method for obtaining the pose features includes:

[0070] 1) The keypoint sequence or keypoint auxiliary data is three-dimensionally compressed based on the second variational autoencoder to obtain a first sequence with the same size as the noise vector. In this embodiment, the second variational autoencoder is the same as the first variational autoencoder, and will not be described again here. The noise vector is latent noise sampled from a standard Gaussian distribution.

[0071] 2) Based on the pose adapter module, feature extraction is performed on the first sequence to obtain a first feature with the same size as the noise vector. In this embodiment, the pose adapter module is a lightweight 3D convolutional network, including a series of convolutional layers and corresponding activation functions and normalization layers. The extracted pose features have the same spatial size as the input noise.

[0072] 3) Add the first feature to the noise vector element-wise to obtain the pose feature. Element-wise addition refers to adding elements at the same position in different vectors. For example, assuming that the pose feature or the noise vector each has multiple elements, the pose feature is represented as (A, B, C, D), and the noise vector is represented as (a, b, c, d). Then, element-wise addition requires adding the first element A in the pose feature to the first element a in the noise vector, adding the second element B to b, adding the third element C to c, and adding the fourth element D to d. The resulting pose feature contains the noise vector needed for the diffusion model to generate new data.

[0073] Based on the above technical means, the corresponding pose features in the video can be extracted through the key point sequence. The pose features are used to drive the generated video frames, ensuring that the content in the generated video can perform the corresponding actions. The key point sequence is compressed, including the time dimension, so that the obtained pose features can fully take into account the temporal dynamics and enhance the coherence of the generated video.

[0074] In this embodiment of the application, the method for obtaining text features includes:

[0075] Expand the input text prompts or auxiliary text data to obtain a refined text description;

[0076] Then the refined text description is converted into a hidden representation;

[0077] The hidden representation is decoded to generate a text embedding vector, and the resulting text embedding vector is used as a text feature.

[0078] In some feasible embodiments, a pre-trained Large Language Model (LLM) is used to expand the input text prompts or text auxiliary data to obtain a refined text description. A T5 model is then used to extract features from this refined text description. The encoder in the T5 model includes multiple self-attention layers and feedforward layers. Through these layers, the refined text description is converted into a set of hidden representations. The decoder in the T5 model then processes these hidden representations step-by-step to generate text embedding vectors. The decoder includes multiple self-attention layers, cross-attention layers, and feedforward layers. The text embedding vectors are the text features extracted based on the text prompt information.

[0079] Therefore, the key information in the text prompts is expanded to output a more concise text description. The text embedding vector is extracted based on the concise text description, which can better reflect the key features in the text prompt information.

[0080] Therefore, multimodal reference features can be extracted using the aforementioned method.

[0081] S3: Generate the target video corresponding to the reference data based on the multimodal features.

[0082] In some feasible embodiments, generating the target video corresponding to the reference data based on the multimodal features includes:

[0083] The spatiotemporal information of the multimodal features is calculated using a backbone network; the backbone network is constructed based on a DIT diffusion network, and the spatiotemporal information is used to characterize the content displayed in the video frame and the interaction between video frames.

[0084] The spatiotemporal information is normalized to obtain variance prediction. In this embodiment, a normalization layer is used to normalize the spatiotemporal information.

[0085] The variance prediction is decoded in pixel space to obtain the target video. In some feasible embodiments, a pre-trained Slice Variational Auto-Encoder (Slice VAE) is used to decode the variance prediction into pixel space. The Slice VAE divides the video into multiple independent parts along the temporal dimension, and each part is encoded and decoded separately. For the video, during the decoding stage, features are concatenated with features from preceding and following parts to achieve feature consistency. The Slice VAE can generate 64 frames of video at a time, and can encode longer, higher-quality face videos.

[0086] Based on the above technical means, by using a DIT-based diffusion network to calculate the spatiotemporal information of multimodal features, information with more coherent temporal relationships and spatial relationships that are more in line with natural laws can be extracted from multimodal features. After normalization processing, variance prediction is obtained. The interaction between frames in the target video generated based on variance prediction is more accurate, and the spatial relationship between the content in each frame of the video is more reasonable, thus improving the video quality.

[0087] In some feasible embodiments, the step of using a backbone network to calculate the spatiotemporal information of the multimodal features includes:

[0088] The text features are used as the key and value inputs for each denoised spatial attention block and each denoised temporal attention block. The output of the previous denoised spatial attention block is used as the query input for the next denoised temporal attention block. The output of the previous denoised temporal attention block and the image features are added element-wise as the query input for the next denoised spatial attention block. The first denoised spatial attention block is located at the beginning of the backbone network, and the query input for the first denoised spatial attention block is the output of the pose features and the image features added element-wise.

[0089] It should be noted that in this application, the image features are spatial attention features derived from multiple feature extraction spatial attention blocks. Therefore, when superimposing image features, the query input of each denoising spatial attention block needs to be superimposed based on its position among all denoising spatial attention blocks to determine the corresponding spatial features output by the feature extraction spatial attention block. For example, the query input of the first denoising spatial attention block is the output of superimposing the spatial features output by the first feature extraction spatial attention block with the pose features; the query input of the second denoising spatial attention block is the output of superimposing the spatial features output by the second feature extraction spatial attention block with the output of the first denoising temporal attention block; the query input of the third denoising spatial attention block is the output of superimposing the spatial features output by the third feature extraction spatial attention block with the output of the second denoising temporal attention block, and so on.

[0090] Based on the aforementioned technical methods, videos are generated using multiple reference data, supporting multi-condition guided face video generation. Image features are extracted using a diffusion network, and video generation is also achieved using a diffusion network, realizing video generation based on a dual diffusion network. The denoising temporal attention block can extract temporal relationship information from multimodal features, while the denoising spatial attention block can extract information whose spatial relationships more conform to natural laws. This information can accurately reflect the interaction between frames in subsequent videos and the spatial relationship between the content in each frame, thereby improving the quality of the generated video and avoiding frames with abrupt changes in key features.

[0091] The video generation method of this application is based on a video generation model. The video generation method of this application will be explained below in conjunction with the structure of the video generation model.

[0092] like Figure 2 As shown, the video generation model of this application includes a multimodal reference feature extraction part and a video generation part. The multimodal reference feature extraction part includes a text feature extraction module, a pose feature extraction module, and an image feature extraction module based on a diffusion model. The video generation part includes a backbone network based on a DIT model and a decoding module.

[0093] The image feature extraction module is configured to extract image features from a reference image or image auxiliary data.

[0094] The text feature extraction module is configured to extract text features from text prompts or text auxiliary data.

[0095] The pose feature extraction module is configured to extract pose features from keypoint sequences or keypoint auxiliary data.

[0096] In some feasible embodiments, in order to obtain corresponding auxiliary data, the text feature extraction module is further configured to obtain text auxiliary data for extracting text features when no text prompt information is input. In one feasible embodiment, after obtaining reference data, the text feature extraction module can determine whether the reference data contains text prompt information. If it does not contain text prompt information, it obtains text auxiliary data for extracting text features. The text features extracted using the text auxiliary data are all zero text features.

[0097] The pose feature extraction module is further configured to acquire keypoint auxiliary data for pose feature extraction when no keypoint sequence is input. In one feasible embodiment, after acquiring reference data, the pose feature extraction module can determine whether the reference data contains a keypoint sequence. If it does not contain a keypoint sequence, it acquires keypoint auxiliary data for pose feature extraction. The pose feature extracted using the keypoint auxiliary data is the result of adding the all-zero pose feature to the noise vector element-wise. That is, the pose feature obtained after feature extraction using the keypoint auxiliary data is equal to the noise vector.

[0098] The image feature extraction module is further configured to acquire image auxiliary data for extracting image features when no reference image is input. In one feasible embodiment, after acquiring the reference data, the image feature extraction module can determine whether the reference data contains a reference image. If it does not contain a reference image, it acquires image auxiliary data for extracting image features, and the image features extracted using the image auxiliary data are all zero image features.

[0099] Based on the above technical means, when the reference data includes one or two of the following: reference image, key point sequence, and text prompt information, the corresponding auxiliary data is used to replace the unprovided reference data in order to obtain the corresponding data features, ensuring that the video is generated normally, while the auxiliary data does not affect the video generation result.

[0100] In some feasible embodiments, such as Figure 2As shown, the image feature extraction module includes a first variational autoencoder (VAE_Enc), a first embedding layer (embed to tokens), and a spatial transformation module (Transformer block) based on a diffusion model. The spatial transformation module based on a diffusion model includes only multiple feature extraction spatial attention blocks.

[0101] Extracting image features from a reference image or image-aided data, including:

[0102] 1) Perform three-dimensional compression on the reference image or image auxiliary data based on the first variational autoencoder to obtain the variational probability distribution of the reference image or the image auxiliary data.

[0103] 2) Based on the first embedding layer, the variational probability distribution is represented as a series of tokens.

[0104] 3) such as Figure 3 As shown, text features are used as key and value inputs for each feature extraction spatial attention block, and the output of the previous feature extraction spatial attention block is used as the query input for the next feature extraction spatial attention block. The spatial attention features of different feature extraction spatial attention blocks are calculated progressively, and the spatial attention features of multiple feature extraction spatial attention blocks are used as image features; wherein, the query input of the first feature extraction spatial attention block is the series of flags.

[0105] The extracted image features are stored in a list (bank) for subsequent video generation.

[0106] In some feasible embodiments, such as Figure 2 As shown, the pose feature extraction module includes a second variational autoencoder (VAE_Enc), a pose adaptation module, and an overlay module;

[0107] The extraction of pose features from keypoint sequences or keypoint auxiliary data includes:

[0108] 1) Based on the second variational autoencoder, the key point sequence or key point auxiliary data is three-dimensionally compressed to obtain a first sequence with the same size as the noise vector.

[0109] 2) Based on the pose adaptation module, feature extraction is performed on the first sequence to obtain pose features of the same size as the noise vector.

[0110] 3) The pose features are added element-wise to the noise vector based on the overlay module to obtain the pose features.

[0111] In some feasible embodiments, extracting text features from text prompts or text auxiliary data includes:

[0112] Expand the input text prompts or auxiliary text data to obtain a refined text description;

[0113] Then the refined text description is converted into a hidden representation;

[0114] The hidden representation is decoded to generate a text embedding vector, and the resulting text embedding vector is used as a text feature.

[0115] In some feasible embodiments, a pre-trained Large Language Model (LLM) is used to expand the input text prompts or text auxiliary data to obtain a refined text description. A T5 model is then used to extract features from this refined text description. The encoder in the T5 model includes multiple self-attention layers and feedforward layers. Through these layers, the refined text description is converted into a set of hidden representations. The decoder in the T5 model then processes these hidden representations step-by-step to generate text embedding vectors. The decoder includes multiple self-attention layers, cross-attention layers, and feedforward layers. The text embedding vectors are the text features extracted based on the text prompt information.

[0116] Therefore, the key information in the text prompts is expanded to output a more concise text description. The text embedding vector is extracted based on the concise text description, which can better reflect the key features in the text prompt information.

[0117] The backbone network is configured to calculate spatiotemporal information based on multimodal features. The obtained spatiotemporal information is normalized by a normalization layer and then converted into variance prediction by a fully connected layer. This spatiotemporal information is used to characterize the content displayed in video frames and the interactions between video frames. In this embodiment, the backbone network includes a spatiotemporal transformation module (Transformerblock), a normalization layer, and a fully connected layer. Multiple spatiotemporal transformation modules are included, comprising cross-connected spatial attention modules and temporal attention modules, used to calculate spatiotemporal information based on multimodal features. The normalization layer normalizes the spatiotemporal information extracted by the spatiotemporal transformation modules, and the fully connected layer converts the normalized spatiotemporal information into variance prediction. This allows variance prediction of the target data to be generated based on reference features, and decoding these variance predictions into pixel space can generate a video with clear features.

[0118] The decoding module is used to decode the variance prediction into pixel space to obtain the video. In some feasible embodiments, a pre-trained Slice Variational Auto-Encoder (Slice VAE) is used to decode the variance prediction into pixel space. The Slice VAE divides the video into multiple independent parts along the time dimension, and each part is encoded and decoded separately. For the video, in the decoding stage, the features are concatenated with the features before and after to achieve feature consistency. The Slice VAE can generate 64 frames of video at a time, and can encode longer, higher-quality face videos.

[0119] In some feasible embodiments, the backbone network includes cross-connected denoised spatial attention blocks and denoised temporal attention blocks, which constitute a series-connected spatiotemporal transformation module. The spatiotemporal transformation module calculates spatiotemporal information based on reference features, such as... Figure 4 As shown, it specifically includes:

[0120] The image features are used as the key and value inputs for each denoised spatial attention block and each denoised temporal attention block. The output of the previous denoised spatial attention block is used as the query input for the next denoised temporal attention block. The output of the previous denoised temporal attention block and the image features are added element-wise as the query input for the next denoised spatial attention block. The first denoised spatial attention block is located at the beginning of the backbone network, and the query input for the first denoised spatial attention block is the output of the pose features and the image features added element-wise.

[0121] In practical applications, the backbone network also includes a second embedding module. The second embedding module represents the pose features as a series of symbols and then inputs them into the spatiotemporal transformation module to calculate the spatiotemporal attention information. In some feasible embodiments, the pose features are segmented using an embed totokens layer to represent the input pose features as a series of symbols.

[0122] Based on the above technical means, videos are generated using multiple reference data, supporting multi-condition guided face video generation. The dual diffusion model realizes video generation, which can achieve better face preservation effect, avoid face mutation frames in the generated face video, and improve the quality of face video.

[0123] In some feasible embodiments, a high-quality self-collected Asian face dataset, particularly East Asian face data, is used to train the model. This ensures the high quality and accuracy of the model in generating Asian face videos, improving upon the shortcomings of models trained using open-source datasets in generating Asian face videos and achieving superior video generation performance. Furthermore, the backbone denoising network of this application can serve as a foundation for subsequent tasks, including but not limited to audio-driven face video generation, cartoon face generation, and face stylization, offering significant advantages.

[0124] A second aspect of this application provides a video generation apparatus, such as... Figure 5 As shown, the video generation device includes:

[0125] The data acquisition unit is used to acquire reference data;

[0126] The feature extraction unit is used to obtain multimodal features based on the reference data; the multimodal reference features include: image features, text features, and pose features;

[0127] The video generation unit is used to generate a target video corresponding to the reference data based on the multimodal features.

[0128] Based on the aforementioned technical means, the device generates target videos based on multimodal features including image features, text features, and pose features. The multi-condition driving of target video generation allows the extracted multimodal features to reflect more realistic video content, making the generated video content more lifelike and having better feature preservation effects, avoiding abrupt frame changes, and improving video quality.

[0129] In some feasible embodiments, obtaining multimodal features based on the reference data includes:

[0130] Based on the reference data and auxiliary data, image features, text features, and pose features corresponding to the reference data are obtained; the reference data includes one or more of the following: reference image, key point sequence, and text prompt information; the auxiliary data includes one or more of the following: image auxiliary data, text auxiliary data, and key point auxiliary data.

[0131] Based on the above technical means, when the reference data only includes one or two of the following: reference image, key point sequence, and text prompt information, the corresponding auxiliary data is used to replace the unprovided reference data in order to obtain the corresponding data features, ensuring the normal progress of multi-condition driven video generation. At the same time, the auxiliary data does not affect the video generation result, and the generated video has a high degree of consistency with the input reference data.

[0132] To acquire multimodal features, the feature extraction unit includes an image feature extraction module, a text feature extraction module, and a pose feature extraction module. The image feature extraction module is configured to extract image features from a reference image or image-assisted data. The text feature extraction module is configured to extract text features from text prompts or text-assisted data. The pose feature extraction module is configured to extract pose features from keypoint sequences or keypoint-assisted data.

[0133] The image feature extraction module is constructed based on a diffusion network of DIT. The image feature extraction module includes a first variational autoencoder, a first embedding layer, and a spatial transformation module based on the diffusion model. The spatial transformation module based on the diffusion model includes only multiple feature extraction spatial attention blocks.

[0134] The first variational autoencoder is used to perform three-dimensional compression on the reference image or image-aided data to obtain the variational probability distribution of the reference image or image-aided data.

[0135] The first embedding layer is used to represent the variational probability distribution as a series of tokens.

[0136] The spatial transformation module is used for spatial attention feature extraction, specifically including:

[0137] Text features are used as key and value inputs for each feature extraction spatial attention block, and the output of the previous feature extraction spatial attention block is used as the query input for the next feature extraction spatial attention block. The spatial attention features of different feature extraction spatial attention blocks are calculated progressively, and the spatial attention features of multiple feature extraction spatial attention blocks are used as image features; wherein, the query input of the first feature extraction spatial attention block is the series of flags.

[0138] Based on the above technical means, the face image is compressed including the time dimension so that the obtained image features can fully consider the time dynamics. A series of tags are used to drive the spatial attention calculation of text features so that the obtained image features are more consistent with the text features. By performing spatial attention calculation on the variational probability distribution of the reference image after time dimension compression, the obtained image features have both time dimension information and spatial dimension information, which improves the feature accuracy and coherence of the generated video.

[0139] The pose feature extraction module includes a second variational autoencoder, a pose adaptation module, and an overlay module.

[0140] The second variational autoencoder is used to perform three-dimensional compression on the keypoint sequence or keypoint auxiliary data to obtain a first sequence with the same size as the noise vector.

[0141] The pose adaptation module is used to extract features from the first sequence to obtain a first feature with the same size as the noise vector;

[0142] The overlay module is used to add the first feature to the noise vector element by element to obtain the pose feature.

[0143] Based on the above technical means, the corresponding pose features in the video can be extracted through the key point sequence. The pose features are used to drive the generated video frames, ensuring that the content in the generated video can perform the corresponding actions. The key point sequence is compressed, including the time dimension, so that the obtained pose features can fully take into account the temporal dynamics and enhance the coherence of the generated video.

[0144] In this embodiment, the backbone network includes a spatiotemporal transformation module (Transformer block), a normalization layer, and a fully connected layer. The spatiotemporal transformation module comprises multiple modules, including cross-connected spatial attention and temporal attention modules, used to calculate spatiotemporal information based on multimodal features. The normalization layer normalizes the spatiotemporal information extracted by the spatiotemporal transformation module, and the fully connected layer converts the normalized spatiotemporal information into variance prediction. This allows the generation of variance predictions for the target data based on reference features, and decoding these variance predictions into pixel space can generate a video with clear features.

[0145] The decoding module is used to decode the variance prediction into pixel space to obtain the video. In some feasible embodiments, a pre-trained Slice Variational Auto-Encoder (Slice VAE) is used to decode the variance prediction into pixel space. The Slice VAE divides the video into multiple independent parts along the time dimension, and each part is encoded and decoded separately. For the video, in the decoding stage, the features are concatenated with the features before and after to achieve feature consistency. The Slice VAE can generate 64 frames of video at a time, and can encode longer, higher-quality face videos.

[0146] In some feasible embodiments, the backbone network includes cross-connected denoised spatial attention blocks and denoised temporal attention blocks, which constitute a series-connected spatiotemporal transformation module. The spatiotemporal transformation module calculates spatiotemporal information based on reference features, specifically including:

[0147] The image features are used as the key and value inputs for each denoised spatial attention block and each denoised temporal attention block. The output of the previous denoised spatial attention block is used as the query input for the next denoised temporal attention block. The output of the previous denoised temporal attention block and the image features are added element-wise as the query input for the next denoised spatial attention block. The first denoised spatial attention block is located at the beginning of the backbone network, and the query input for the first denoised spatial attention block is the output of the pose features and the image features added element-wise.

[0148] A third aspect of this application provides an electronic device, comprising:

[0149] The memory is configured to store instructions; and

[0150] The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the video generation method.

[0151] A fourth aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform the video generation method.

[0152] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the video generation method.

[0153] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0158] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0159] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0160] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0161] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A video generation method, characterized in that, The video generation method includes: Obtain reference data; Based on the reference data, multimodal features are obtained; the multimodal reference features include: image features, text features, and pose features; Based on the multimodal features, a target video corresponding to the reference data is generated.

2. The video generation method according to claim 1, characterized in that, The step of obtaining multimodal features based on the reference data includes: Based on the reference data and auxiliary data, image features, text features, and pose features corresponding to the reference data are obtained; the reference data includes one or more of the following: reference image, key point sequence, and text prompt information; the auxiliary data includes one or more of the following: image auxiliary data, text auxiliary data, and key point auxiliary data.

3. The video generation method according to claim 2, characterized in that, The image features are generated based on a DIT diffusion network; the method for obtaining the image features includes: The reference image or image auxiliary data is three-dimensionally compressed based on the first variational autoencoder to obtain the variational probability distribution of the reference image or the image auxiliary data. The variational probability distribution is converted into a series of labels; Text features are used as key and value inputs for each feature extraction spatial attention block, and the output of the previous feature extraction spatial attention block is used as the query input for the next feature extraction spatial attention block. The spatial attention features of different feature extraction spatial attention blocks are calculated progressively, and the spatial attention features of multiple feature extraction spatial attention blocks are used as image features; wherein, the query input of the first feature extraction spatial attention block is the series of flags.

4. The video generation method according to claim 2, characterized in that, The method for obtaining the pose features includes: Based on the second variational autoencoder, the key point sequence or key point auxiliary data is three-dimensionally compressed to obtain a first sequence with the same size as the noise vector. Based on the pose adaptation module, feature extraction is performed on the first sequence to obtain a first feature with the same size as the noise vector; The first feature is added element-wise to the noise vector to obtain the pose feature.

5. The video generation method according to claim 1, characterized in that, The step of generating the target video corresponding to the reference data based on the multimodal features includes: The spatiotemporal information of the multimodal features is calculated using a backbone network; the backbone network is constructed based on a DIT diffusion network. The spatiotemporal information is normalized to obtain variance prediction; The variance prediction is decoded in pixel space to obtain the target video.

6. The video generation method according to claim 5, characterized in that, The calculation of the spatiotemporal information of the multimodal features using a backbone network includes: The text features are used as the key and value inputs for each denoising spatial attention block and each denoising temporal attention block. The output of the previous denoising spatial attention block is used as the query input for the next denoising temporal attention block. The output of the previous denoising temporal attention block is added element-wise with the image features, and the result is used as the query input for the next denoising spatial attention block. The first denoising spatial attention block is located at the beginning of the backbone network, and the query input for the first denoising spatial attention block is the output of the pose features added element-wise with the image features.

7. A video generation apparatus, characterized in that, The video generation device includes: The data acquisition unit is used to acquire reference data; The feature extraction unit is used to obtain multimodal features based on the reference data; the multimodal reference features include: image features, text features, and pose features; The video generation unit is used to generate a target video corresponding to the reference data based on the multimodal features.

8. An electronic device, characterized in that, include: The memory is configured to store instructions; as well as The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the video generation method of any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the video generation method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video generation method according to any one of claims 1 to 6.

Images