A video generation method and related apparatus

Abstract

The application discloses a video generation method and related device, and relates to the field of video generation, and comprises the following steps: performing face detection on a target image containing a target object to obtain a face region image; performing face coding on the face region image to obtain face features, performing image coding on the face region image to obtain image features; inputting the face features and the image features into a deep learning network model; and performing denoising processing on noise data generated by the deep learning network model to obtain a video containing the target object. The application can achieve the purpose of maintaining identity consistency.

Description

Technical Field

[0001] This application relates to the field of video generation technology, and in particular to a video generation method and related apparatus. Background Technology

[0002] With the development of digital human video generation technology, identity consistency has become a significant challenge. Identity consistency refers to ensuring that, regardless of changes in the input driving data (such as audio and text), the target person in the generated video maintains the same identity characteristics as the original person throughout the video synthesis process. This means ensuring that the appearance, facial features, and other identity characteristics of the generated person remain stable and unchanged. Current digital human video generation methods add extra control modules to ensure that the generated video conforms to the user's posture, style, and movement requirements. While this allows for customization of specific actions or styles, it often neglects the control of identity information. Summary of the Invention

[0003] In view of the above problems, this application provides a video generation method and related apparatus to achieve the purpose of maintaining identity consistency. The specific solution is as follows:

[0004] The first aspect of this application provides a video generation method, including:

[0005] Acquire a target image; wherein the target image contains a target object;

[0006] Face detection is performed on the target image to obtain a face region image;

[0007] The face region image is subjected to face encoding to obtain face features, and the face region image is subjected to image encoding to obtain image features;

[0008] The facial features and image features are input into a deep learning network model, and the noise data generated by the deep learning network model is denoised to obtain a video containing the target object.

[0009] In one possible implementation, before inputting the facial features and the image features into the deep learning network model, the method further includes:

[0010] Get text prompts;

[0011] Based on the text prompt information, obtain appearance prompt text and action prompt text;

[0012] The appearance prompt text is encoded to obtain appearance features, and the action prompt text is encoded to obtain action features;

[0013] The step of inputting the facial features and image features into a deep learning network model, and performing denoising processing on the noisy data generated by the deep learning network model to obtain a video containing the target object includes:

[0014] The facial features, image features, appearance features, and action features are input into the deep learning network model, and the noisy data generated by the deep learning network model is denoised to obtain a video containing the target object.

[0015] In one possible implementation, the step of inputting the facial features, image features, appearance features, and action features into the deep learning network model, and performing denoising processing on the noisy data generated by the deep learning network model to obtain a video containing the target object includes:

[0016] The facial features, image features, appearance features, action features, and noise data are input into different processing layers of the current level of the deep learning network model for processing.

[0017] After obtaining the processing results of each processing layer, the processing results of each processing layer are input into the next layer of the deep learning network model for processing, until the last layer of the deep learning network model is processed to obtain the video containing the target object.

[0018] In one possible implementation, the step of inputting the facial features, image features, appearance features, action features, and noise data into different processing layers of the current level of the deep learning network model for processing includes:

[0019] The noisy data is input into the convolutional layer of the current layer of the deep learning network model for processing to obtain the feature map of the current layer;

[0020] The facial features and the image features are input into the first cross-attention layer of the current layer of the deep learning network model for processing to obtain the first attention weight of the current layer;

[0021] The appearance features are input into the second cross-attention layer of the current layer of the deep learning network model for processing to obtain the second attention weights of the current layer;

[0022] The action features are input into the third cross-attention layer of the current layer of the deep learning network model for processing to obtain the third attention weight of the current layer.

[0023] In one possible implementation, after obtaining the processing results of each processing layer, inputting the processing results of each processing layer into the next level of the deep learning network model for further processing includes:

[0024] The feature map of the current layer, the first attention weight of the current layer, the second attention weight of the current layer, and the third attention weight of the current layer are input into the convolutional layer of the next layer of the deep learning network model for processing to obtain the feature map of the next layer.

[0025] The first attention weight of the current level is input into the first cross-attention layer of the next level of the deep learning network model for processing to obtain the first attention weight of the next level.

[0026] The second attention weight of the current layer is input into the second cross-attention layer of the next layer of the deep learning network model for processing to obtain the second attention weight of the next layer;

[0027] The third attention weight of the current layer is input into the second cross-attention layer of the next layer of the deep learning network model for processing to obtain the third attention weight of the next layer.

[0028] In one possible implementation, the step of inputting the facial features and the image features into the first cross-attention layer of the current layer of the deep learning network model for processing to obtain the first attention weights of the current layer includes:

[0029] A first query matrix, a first key matrix, and a first value matrix are generated based on the facial features and the image features;

[0030] Multi-head attention calculation is performed based on the first query matrix, the first key matrix, and the first value matrix to obtain the first attention weight of the current level;

[0031] The step of inputting the appearance features into the second cross-attention layer of the current layer of the deep learning network model for processing to obtain the second attention weights of the current layer includes:

[0032] A second query matrix, a second key matrix, and a second value matrix are generated based on the appearance features;

[0033] Attention is calculated based on the second query matrix, the second key matrix, and the second value matrix to obtain the second attention weight of the current level;

[0034] The step of inputting the action features into the third cross-attention layer of the current layer of the deep learning network model for processing to obtain the third attention weights of the current layer includes:

[0035] Generate a third query matrix, a third key matrix, and a third value matrix based on the action features;

[0036] Attention is calculated based on the third query matrix, the third key matrix, and the third value matrix to obtain the third attention weight of the current level.

[0037] A second aspect of this application provides a video generation system, comprising:

[0038] An image acquisition module is used to acquire a target image; wherein the target image contains a target object;

[0039] The face detection module is used to perform face detection on the target image to obtain a face region image;

[0040] The first encoding module is used to perform face encoding on the face region image to obtain face features, and to perform image encoding on the face region image to obtain image features;

[0041] The denoising module is used to input the facial features and the image features into a deep learning network model, and to denoise the noise data generated by the deep learning network model to obtain a video containing the target object.

[0042] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the video generation method described in the first aspect or any implementation thereof.

[0043] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0044] The memory is used to store computer programs;

[0045] The processor is used to execute the computer program so that the electronic device can implement the video generation method of the first aspect or any implementation thereof.

[0046] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the video generation method described in the first aspect or any implementation thereof.

[0047] By employing the aforementioned technical solutions, the video generation method and related apparatus provided in this application consider the identity characteristics of the target object during the video generation process. Face encoding and image encoding are performed on the face region image separately to obtain face features and image features. These face features and image features are then input into a deep learning network model. Noise processing is performed on the noisy data generated by the deep learning network model to obtain a video that conforms to the identity characteristics of the target object, thereby achieving the purpose of maintaining identity consistency. This application performs face encoding and image encoding separately, decoupling identity information and non-identity information in the image, thereby extracting complete identity features without learning other image features unrelated to identity features, and thus preserving the identity information of the target object in the generated video. Attached Figure Description

[0048] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0049] Figure 1 A flowchart of a video generation method provided in this application;

[0050] Figure 2 Flowchart of another video generation method provided in this application;

[0051] Figure 3 A schematic diagram of a video generation method provided in this application;

[0052] Figure 4 A structural diagram of a video generation system provided in this application;

[0053] Figure 5 Another video generation system architecture diagram provided in this application;

[0054] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0055] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0056] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] This application provides a video generation method. The video generation method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0059] This application provides a video generation method, such as... Figure 1 As shown, the method includes:

[0060] Step 101: Obtain the target image; wherein the target image contains the target object.

[0061] Step 102: Perform face detection on the target image to obtain the face region image.

[0062] Step 103: Perform face encoding on the face region image to obtain face features, and perform image encoding on the face region image to obtain image features.

[0063] Step 104: Input facial features and image features into the deep learning network model, and perform denoising processing on the noisy data generated by the deep learning network model to obtain a video containing the target object.

[0064] The target image is an image containing the target object. For generating digital human videos, the target object can be a person, and the image containing the target object can display the person's face for subsequent extraction of identity features. These identity features can include facial features, physical appearance features, etc.

[0065] After performing face detection on the target image and obtaining the face region image, face encoding and image encoding are performed on the face region image to obtain face features and image features respectively. In practical applications, a face encoder can be used to encode the face region image, specifically the ArcFace model. The ArcFace model uses an angle embedding-based loss function to enhance the distinguishability between different identities while maintaining the similarity of the same identity. By shifting the angle between the input features and the classification center, ArcFace can better distinguish the facial features of different people in the feature space. In practical applications, an image encoder can be used to encode the face region image, specifically the CLIP (Contrastive Language-Image Pre-Training) image encoder. The CLIP image encoder is the image encoding part of the CLIP model, designed to convert the image into a high-dimensional feature vector. The CLIP image encoder first feeds the preprocessed image into a convolutional neural network. Through multiple layers of convolution and pooling operations, it gradually extracts features at different levels of the image (such as edges, shapes, colors, and complex semantic objects). After a series of convolution and pooling layers, the image is represented as a high-dimensional feature vector, which contains the main objects, scenes, and semantic information in the image.

[0066] After obtaining facial and image features, these features are input into a deep learning network model. The model then denoises the generated noisy data to produce a video containing the target object. This deep learning network model is a pre-trained model, and the noisy data can conform to a Gaussian distribution, or it can be a noisy video composed of multiple noisy images. During the denoising process, the model uses facial and image features to guide the generation of a video that matches the identity of the target person, thus maintaining identity consistency after denoising.

[0067] In one possible implementation, the deep learning network model in step 104 is obtained through training, and the training method of this model specifically includes:

[0068] Acquire the training image and the training video; the training image is an image containing the target object, and the training video is a video generated by adding noise to a video containing the target object. Perform face detection on the training image to obtain the training face region image. Input the face features obtained by face encoding of the training face region image, the image features obtained by image encoding of the training face region image, and the training video into the deep learning network model to be trained for denoising training, and obtain the trained deep learning network model.

[0069] The training image is an image containing the target object. For generating digital human videos, the target object can be a person, and the image containing the target object can display the person's face for subsequent extraction of identity features. These identity features can include facial features, appearance features, etc. There can be multiple training images, which can include unobstructed frontal images of the face and everyday images of the target person from different angles and in different styles. For example, if there are four training images, one image can be an unobstructed frontal image of the target person's face, and the other three images can be everyday images of the target person from different angles and in different styles.

[0070] The training video is generated by adding noise to a video containing a target object. When adding noise to the video containing the target object, a diffusion model can be used. This noise-adding process is a fixed Markov chain used to add noise to the original data, transforming it into Gaussian data. This is achieved by adding noise to each frame of the video containing the target object at each time step. Adding noise yields a series of intermediate states. After T noise addition operations, the image It will become a pure Gaussian noise map that conforms to a standard normal distribution, thus obtaining the video to be trained.

[0071] After performing face detection on the training images to obtain the face region images to be trained, face encoding and image encoding are performed on the face region images to be trained. In practical applications, a face encoder can be used to encode the face region images, specifically a pre-trained ArcFace model. The ArcFace model uses an angle embedding-based loss function to enhance the discriminability between different identities while maintaining the similarity of the same identity. By shifting the angle between the input features and the classification center, ArcFace can better distinguish the facial features of different people in the feature space. In practical applications, an image encoder can be used to encode the face region images, specifically a CLIP (Contrastive Language-Image Pre-Training, Multimodal Pre-trained Neural Network) image encoder. The CLIP image encoder is the image encoding part of the CLIP model, designed to convert images into high-dimensional feature vectors. The CLIP image encoder first feeds the preprocessed image into a convolutional neural network. Through multiple layers of convolution and pooling operations, it gradually extracts features at different levels of the image (such as edges, shapes, colors, and complex semantic objects). After a series of convolution and pooling layers, the image is represented as a high-dimensional feature vector, which contains the main objects, scenes, and semantic information in the image.

[0072] After obtaining facial and image features, these features, along with the training video, are input into a deep learning network model for denoising training, resulting in a trained deep learning network model. During the denoising process of the training video, the facial and image features guide the model to learn the identity characteristics of the target person. Once trained, the model can generate videos that match the identity characteristics of the target person, thus achieving the goal of maintaining identity consistency.

[0073] This application also provides a video generation method, which can be referred to. Figure 2 . Figure 2 A video generation method provided in this application includes:

[0074] Step 201: Obtain the target image; wherein the target image contains the target object.

[0075] Step 202: Perform face detection on the target image to obtain the face region image.

[0076] Step 203: Perform face encoding on the face region image to obtain face features, and perform image encoding on the face region image to obtain image features.

[0077] Step 204: Obtain text prompt information.

[0078] Step 205: Obtain appearance prompt text and action prompt text based on text prompt information.

[0079] Step 206: Encode the appearance prompt text to obtain appearance features, and encode the action prompt text to obtain action features.

[0080] Step 207: Input facial features, image features, appearance features, and action features into the deep learning network model, and perform noise reduction processing on the noisy data generated by the deep learning network model to obtain a video containing the target object.

[0081] Steps 201-203 and Figure 1 Steps 101-103 shown are similar and will not be repeated here.

[0082] To better match user commands with the generated video containing the target object, text prompts can be input. Based on these prompts, appearance and action prompts can be obtained. In practical applications, a large language model can be used to split the text prompts into appearance and action prompts. A text encoder can be used to encode the appearance prompts to obtain appearance features, and a text encoder can be used to encode the action prompts to obtain action features. Optionally, the text encoder can be a T5 model.

[0083] like Figure 3 As shown, Figure 3 This is a schematic diagram of a video generation method provided in an embodiment of this application. A person's image is input into a face encoder and an image encoder for feature extraction. The resulting face features and image features are then input into a deep learning network model. Text prompts are split into appearance prompts and action prompts. The appearance prompts are encoded using a text encoder to obtain appearance features, and the action prompts are encoded using a text encoder to obtain action features. The appearance and action features are then input into the deep learning network. The deep learning network model uses the face features, image features, appearance features, and action features to denoise the noisy data, thereby generating a person's video.

[0084] In one possible implementation, facial features, image features, appearance features, and motion features are input into a deep learning network model. The noisy data generated by the deep learning network model is then denoised to obtain a video containing the target object, including:

[0085] Facial features, image features, appearance features, action features, and noise data are input into different processing layers of the current level of the deep learning network model for processing;

[0086] After obtaining the processing results of each processing layer, the processing results of each processing layer are input into the next layer of the deep learning network model for processing, until the last layer of the deep learning network model is completed, and a video containing the target object is obtained.

[0087] To enable deep learning network models to better understand the semantic information of the input and provide more accurate denoising guidance, facial features, image features, appearance features, and action features can be decoupled by inserting them into different processing layers. Facial features and image features can be inserted into the same processing layer. This deep learning network model can serve as a denoising model, aiming to remove noise from the model-generated data through multi-level denoising processing to obtain a video containing the target object. The processing result of each layer is input into the next layer for further processing until all layers are completed, resulting in the final video containing the target object. This deep learning network model includes multiple layers, with each layer containing at least four processing layers. One processing layer processes facial features and image features, while the other three processing layers process appearance features, action features, and noise data, respectively, yielding the processing result for each layer. The denoising process can be viewed as a process of predicting noise, and this denoising process can employ the inverse diffusion process of a diffusion model, which is also a Markov process. In practical applications, three cross-attention layers can be added to each layer of the UNet model to obtain the denoising model. After the final layer of the deep learning network model is processed, a video containing the target object can be obtained.

[0088] In one possible implementation, facial features, image features, appearance features, motion features, and noise data are input into different processing layers of the current level of the deep learning network model for processing, including:

[0089] Noisy data is input into the convolutional layer of the current layer of the deep learning network model for processing to obtain the feature map of the current layer;

[0090] The facial features and image features are input into the first cross-attention layer of the current layer of the deep learning network model for processing to obtain the first attention weight of the current layer;

[0091] The appearance features are input into the second cross-attention layer of the current layer of the deep learning network model for processing to obtain the second attention weights of the current layer;

[0092] The action features are input into the third cross-attention layer of the current layer of the deep learning network model for processing, and the third attention weight of the current layer is obtained.

[0093] The deep learning network model can be obtained by adding three cross-attention layers to each layer of the UNet model. Noisy data is input into the convolutional layer of the UNet model for processing to obtain the feature map of the current layer; face features and image features are input into the first cross-attention layer for processing to obtain the first attention weight; appearance features are input into the second cross-attention layer for processing to obtain the second attention weight; and action features are input into the third cross-attention layer for processing to obtain the third attention weight.

[0094] The output of each layer of this deep learning network model for:

[0095]

[0096] In the formula, For the feature map of the current level, As the first attention weight for the current level, It is the sum of the second and third attention weights at the current level.

[0097] In one possible implementation, facial features and image features are input into the first cross-attention layer of the current layer of the deep learning network model for processing, resulting in the first attention weights of the current layer, including:

[0098] Generate a first query matrix, a first key matrix, and a first value matrix based on facial features and image features;

[0099] Multi-head attention calculation is performed based on the first query matrix, the first key matrix, and the first value matrix to obtain the first attention weight of the current level.

[0100] To accurately extract the facial and appearance features of the target person, a multi-head attention formula can be used to calculate the first attention weight. In practical applications, the Query Former query generator can be used to obtain the first attention weight. Query Former is an improvement or extension of the Transformer neural network architecture, commonly used in tasks such as computer vision and natural language processing, especially excelling in object detection and visual question answering. The core idea of ​​Query Former is the introduction of "query" vectors, which are representations learned from the input data and can effectively guide the generation or prediction process. In Query Former, query vectors are generated internally by the deep learning network model, guiding the model to generate appropriate outputs. For facial features and image features, both ID (Identity Document) information related to the target person's facial and appearance features and non-ID information are included. By performing multi-head attention calculation based on the first query matrix, first key matrix, and first value matrix, the model can learn only the identity information related to the target object, without learning non-identity information such as image style and background, thus controlling video generation while maintaining identity consistency.

[0101] Optionally, the appearance features are input into the second cross-attention layer of the current layer of the deep learning network model for processing to obtain the second attention weights of the current layer, including:

[0102] Generate a second query matrix, a second key matrix, and a second value matrix based on appearance features;

[0103] Attention is calculated based on the second query matrix, the second key matrix, and the second value matrix to obtain the second attention weight for the current level.

[0104] Optionally, the action features are input into the third cross-attention layer of the current layer of the deep learning network model for processing to obtain the third attention weights of the current layer, including:

[0105] Generate a third query matrix, a third key matrix, and a third value matrix based on action features;

[0106] Attention is calculated based on the third query matrix, the third key matrix, and the third value matrix to obtain the third attention weight for the current level.

[0107] The following formula can be used to generate the second or third attention weights:

[0108]

[0109] In the formula, For attention weights, For querying the matrix, , The key matrix, , For value matrices, , The query feature is obtained by embedding and transforming appearance or action features. The weight matrix corresponding to the query matrix. For appearance or movement characteristics, This is the weight matrix corresponding to the key matrix. The weight matrix is ​​the matrix corresponding to the value matrix. For vector dimensions.

[0110] In one possible implementation, after obtaining the processing results of each processing layer, the processing results of each processing layer are input into the next layer of the deep learning network model for further processing, including:

[0111] The feature map of the current layer, the first attention weight of the current layer, the second attention weight of the current layer, and the third attention weight of the current layer are input into the convolutional layer of the next layer of the deep learning network model for processing to obtain the feature map of the next layer.

[0112] The first attention weight of the current layer is input into the first cross-attention layer of the next layer of the deep learning network model for processing to obtain the first attention weight of the next layer;

[0113] The second attention weights of the current layer are input into the second cross-attention layer of the next layer of the deep learning network model for processing to obtain the second attention weights of the next layer.

[0114] The third attention weight of the current layer is input into the second cross-attention layer of the next layer of the deep learning network model for processing to obtain the third attention weight of the next layer.

[0115] The feature map of the current layer, the first attention weight of the current layer, the second attention weight of the current layer, and the third attention weight of the current layer are input into the convolutional layer of the next layer of the deep learning network model for processing. That is, the convolutional layer of the next layer of the deep learning network model processes... Process it.

[0116] In one possible implementation, the deep learning network model in step 207 is obtained through training, and the training method of this model specifically includes:

[0117] The process involves acquiring training images and training videos. The training image contains the target object, and the training video is generated by adding noise to a video containing the target object. Face detection is performed on the training image to obtain the training face region image. The face features obtained by face encoding of the training face region image, the image features obtained by image encoding of the training face region image, and the training video are input into the deep learning network model to be trained for denoising and training, resulting in a trained deep learning network model. Text prompts are also acquired for training. Based on these text prompts, appearance prompt text and action prompt text are obtained for training. The appearance features obtained by encoding the appearance prompt text, the action features obtained by encoding the action prompt text, the face features obtained by face encoding of the training face region image, the image features obtained by image encoding of the training face region image, and the training video are input into the deep learning network model to be trained for denoising and training, resulting in a trained deep learning network model.

[0118] To better match user commands with the generated video containing the target object, text prompts can be input during model training. Based on these prompts, appearance and action prompts are obtained. In practical applications, a large language model can be used to split the text prompts into appearance and action prompts. A text encoder can be used to encode the appearance prompts to obtain appearance features, and a text encoder can be used to encode the action prompts to obtain action features. Optionally, the text encoder can be a T5 model.

[0119] The face features obtained by face encoding of the face region image to be trained, the image features obtained by image encoding of the face region image to be trained, the appearance features obtained by encoding the appearance prompt text to be trained, the action features obtained by encoding the action prompt text to be trained, and the video to be trained are input into different processing layers of the current level of the deep learning network model and processed to obtain the processing results of each processing layer.

[0120] Based on the processing results, noise prediction is performed on the current layer of the deep learning network model to obtain the noise prediction results. The model parameters of the deep learning network model are then updated based on the noise prediction results to obtain the updated model.

[0121] If the current level is the last level, then the updated model will be used as the trained deep learning network model.

[0122] If the current level is not the last level, the processing results of each processing layer are input into the updated model to obtain the processing results of the updated model. Then, noise prediction of the current level of the deep learning network model is performed based on the processing results.

[0123] To enable deep learning network models to better understand the semantic information of the input, providing more accurate denoising guidance, and allowing the generated videos to better execute user instructions and accurately extract identity features, facial features, image features, appearance features, and action features can be decoupled by inserting them into different processing layers. This deep learning network model can serve as a denoising model, aiming to restore the training video to a first video containing the target object through multi-level denoising processing. The processing result of each layer is input into the next layer for further processing until all layers are completed, resulting in the final video generation model. This deep learning network model includes multiple layers, with each layer containing at least four processing layers. These four processing layers process facial features, image features, appearance features, action features, and the training video, respectively, yielding the processing result of each layer. The denoising process of the model can be viewed as a process of predicting noise. This denoising process can employ the inverse diffusion process of a diffusion model, learning a denoising model and progressively denoising until the original data is restored. This inverse diffusion process is also a Markov process. In practical applications, three cross-attention layers can be added to each layer of the UNet model to obtain the denoising model.

[0124] The model parameters of the deep learning network model can be updated based on the noise prediction results. The loss function value can be calculated based on the noise prediction results, and the model parameters can be updated using the loss function value.

[0125] The above describes a video generation method provided by an embodiment of this application. The following will describe a system that performs the above video generation method.

[0126] This application provides a video generation system, such as... Figure 4 As shown, the system includes:

[0127] Image acquisition module 401 is used to acquire a target image; wherein the target image contains a target object;

[0128] The face detection module 402 is used to perform face detection on the target image to obtain a face region image;

[0129] The first encoding module 403 is used to perform face encoding on the face region image to obtain face features, and to perform image encoding on the face region image to obtain image features;

[0130] The denoising module 404 is used to input facial features and image features into the deep learning network model, and to denoise the noisy data generated by the deep learning network model to obtain a video containing the target object.

[0131] This application also provides a video generation system, such as... Figure 5As shown, the system includes:

[0132] Image acquisition module 501 is used to acquire a target image; wherein the target image contains a target object;

[0133] The face detection module 502 is used to perform face detection on the target image to obtain a face region image;

[0134] The first encoding module 503 is used to perform face encoding on the face region image to obtain face features, and to perform image encoding on the face region image to obtain image features;

[0135] Information acquisition module 504 is used to acquire text prompt information;

[0136] The text acquisition module 505 is used to obtain appearance prompt text and action prompt text based on text prompt information;

[0137] The second encoding module 506 is used to encode the appearance prompt text to obtain appearance features and to encode the action prompt text to obtain action features.

[0138] The denoising module 507 is used to input facial features, image features, appearance features and action features into the deep learning network model, and to denoise the noisy data generated by the deep learning network model to obtain a video containing the target object.

[0139] In one possible implementation, the denoising module 507 specifically includes:

[0140] The first processing unit is used to input facial features, image features, appearance features, action features, and noise data into different processing layers of the current level of the deep learning network model for processing;

[0141] The second processing unit is used to input the processing results of each processing layer into the next layer of the deep learning network model for processing after obtaining the processing results of each processing layer, until the last layer of the deep learning network model is processed and a video containing the target object is obtained.

[0142] The first processing unit is specifically used for:

[0143] Noisy data is input into the convolutional layer of the current layer of the deep learning network model for processing to obtain the feature map of the current layer;

[0144] The facial features and image features are input into the first cross-attention layer of the current layer of the deep learning network model for processing to obtain the first attention weight of the current layer;

[0145] The appearance features are input into the second cross-attention layer of the current layer of the deep learning network model for processing to obtain the second attention weights of the current layer;

[0146] The action features are input into the third cross-attention layer of the current layer of the deep learning network model for processing, and the third attention weight of the current layer is obtained.

[0147] The second processing unit is specifically used for:

[0148] The feature map of the current layer, the first attention weight of the current layer, the second attention weight of the current layer, and the third attention weight of the current layer are input into the convolutional layer of the next layer of the deep learning network model for processing to obtain the feature map of the next layer.

[0149] The first attention weight of the current layer is input into the first cross-attention layer of the next layer of the deep learning network model for processing to obtain the first attention weight of the next layer;

[0150] The second attention weights of the current layer are input into the second cross-attention layer of the next layer of the deep learning network model for processing to obtain the second attention weights of the next layer.

[0151] The third attention weight of the current layer is input into the second cross-attention layer of the next layer of the deep learning network model for processing to obtain the third attention weight of the next layer.

[0152] Optionally, the first processing unit is also used for:

[0153] Generate a first query matrix, a first key matrix, and a first value matrix based on facial features and image features;

[0154] Multi-head attention calculation is performed based on the first query matrix, the first key matrix, and the first value matrix to obtain the first attention weight of the current level;

[0155] Generate a second query matrix, a second key matrix, and a second value matrix based on appearance features;

[0156] Attention is calculated based on the second query matrix, the second key matrix, and the second value matrix to obtain the second attention weight of the current level.

[0157] Generate a third query matrix, a third key matrix, and a third value matrix based on action features;

[0158] Attention is calculated based on the third query matrix, the third key matrix, and the third value matrix to obtain the third attention weight for the current level.

[0159] This application also provides an electronic device in its embodiments. (See reference...) Figure 6The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0160] like Figure 6 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0161] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0162] This electronic device is capable of implementing the aforementioned video generation method.

[0163] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the video generation methods provided in this application.

[0164] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the video generation methods provided in this application.

[0165] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0167] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0168] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A video generation method, characterized in that, include: Acquire a target image; wherein the target image contains a target object; Face detection is performed on the target image to obtain a face region image; The face region image is subjected to face encoding to obtain face features, and the face region image is subjected to image encoding to obtain image features; The facial features and image features are input into a deep learning network model, and the noise data generated by the deep learning network model is denoised to obtain a video containing the target object.

2. The video generation method according to claim 1, characterized in that, Before inputting the facial features and the image features into the deep learning network model, the method further includes: Get text prompts; Based on the text prompt information, obtain appearance prompt text and action prompt text; The appearance prompt text is encoded to obtain appearance features, and the action prompt text is encoded to obtain action features; The step of inputting the facial features and image features into a deep learning network model, and performing denoising processing on the noisy data generated by the deep learning network model to obtain a video containing the target object includes: The facial features, image features, appearance features, and action features are input into the deep learning network model, and the noisy data generated by the deep learning network model is denoised to obtain a video containing the target object.

3. The video generation method according to claim 2, characterized in that, The step of inputting the facial features, image features, appearance features, and action features into the deep learning network model, and performing denoising processing on the noisy data generated by the deep learning network model to obtain a video containing the target object includes: The facial features, image features, appearance features, action features, and noise data are input into different processing layers of the current level of the deep learning network model for processing. After obtaining the processing results of each processing layer, the processing results of each processing layer are input into the next layer of the deep learning network model for processing, until the last layer of the deep learning network model is processed to obtain the video containing the target object.

4. The video generation method according to claim 3, characterized in that, The step of inputting the facial features, image features, appearance features, action features, and noise data into different processing layers of the current level of the deep learning network model for processing includes: The noisy data is input into the convolutional layer of the current layer of the deep learning network model for processing to obtain the feature map of the current layer; The facial features and the image features are input into the first cross-attention layer of the current layer of the deep learning network model for processing to obtain the first attention weight of the current layer; The appearance features are input into the second cross-attention layer of the current layer of the deep learning network model for processing to obtain the second attention weights of the current layer; The action features are input into the third cross-attention layer of the current layer of the deep learning network model for processing to obtain the third attention weight of the current layer.

5. The video generation method according to claim 4, characterized in that, After obtaining the processing results of each processing layer, the processing results of each processing layer are input into the next level of the deep learning network model for further processing, including: The feature map of the current layer, the first attention weight of the current layer, the second attention weight of the current layer, and the third attention weight of the current layer are input into the convolutional layer of the next layer of the deep learning network model for processing to obtain the feature map of the next layer. The first attention weight of the current level is input into the first cross-attention layer of the next level of the deep learning network model for processing to obtain the first attention weight of the next level. The second attention weight of the current layer is input into the second cross-attention layer of the next layer of the deep learning network model for processing to obtain the second attention weight of the next layer; The third attention weight of the current layer is input into the second cross-attention layer of the next layer of the deep learning network model for processing to obtain the third attention weight of the next layer.

6. The video generation method according to claim 4 or 5, characterized in that, The step of inputting the facial features and the image features into the first cross-attention layer of the current layer of the deep learning network model for processing to obtain the first attention weights of the current layer includes: A first query matrix, a first key matrix, and a first value matrix are generated based on the facial features and the image features; Multi-head attention calculation is performed based on the first query matrix, the first key matrix, and the first value matrix to obtain the first attention weight of the current level; The step of inputting the appearance features into the second cross-attention layer of the current layer of the deep learning network model for processing to obtain the second attention weights of the current layer includes: A second query matrix, a second key matrix, and a second value matrix are generated based on the appearance features; Attention is calculated based on the second query matrix, the second key matrix, and the second value matrix to obtain the second attention weight of the current level; The step of inputting the action features into the third cross-attention layer of the current layer of the deep learning network model for processing to obtain the third attention weights of the current layer includes: Generate a third query matrix, a third key matrix, and a third value matrix based on the action features; Attention is calculated based on the third query matrix, the third key matrix, and the third value matrix to obtain the third attention weight of the current level.

7. A video generation system, characterized in that, include: An image acquisition module is used to acquire a target image; wherein the target image contains a target object; The face detection module is used to perform face detection on the target image to obtain a face region image; The first encoding module is used to perform face encoding on the face region image to obtain face features, and to perform image encoding on the face region image to obtain image features; The denoising module is used to input the facial features and the image features into a deep learning network model, and to denoise the noise data generated by the deep learning network model to obtain a video containing the target object.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the video generation method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the video generation method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the video generation method as described in any one of claims 1 to 6.

Images