Face video generation method, face video model training method, face video model training device and related equipment

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a face video generation method, a face video model training method, a face video model training device and related equipment. The face video generation model training method comprises the steps of obtaining a to-be-processed initial face image, initial audio data and an emotion label; processing the initial face image through a preset 3D face reconstruction tool to obtain first expression feature data; extracting initial audio feature data from the initial audio data, and generating a corresponding emotion single hot feature vector according to the emotion tag; fusing the first expression feature data, the initial audio feature data and the emotion single hot feature vector to obtain a first face expression coefficient; injecting preset noise data into the initial face image to obtain a second face image; and inputting the initial face image, the second face image and the first face expression coefficient to a preset conditional diffusion model to obtain a target face video with expressions, so that the generated face video more accurately expresses the facial expressions corresponding to the emotion tags.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a face video generation method and a model training method, device and related equipment. Background Art

[0002] In the current social media industry, generating accurate and realistic dynamic portrait images has become a vital need, especially in terms of expression control, for example, in a video, a virtual character can show corresponding facial emotional expressions according to the video voice content. Traditional technologies generally combine deep learning and generative adversarial networks (GAN) and other technologies. For example, in generating two-dimensional face images, the generator is responsible for generating images, and the discriminator evaluates whether the generated image is appropriate. For example, the variational autoencoder (VAE) helps learn the potential feature representation of face images and realizes the conversion from random vectors to face images. However, traditional face generation technology often cannot generate facial images with corresponding emotions according to the user's emotions and intentions. Summary of the invention

[0003] The main technical problem solved by the implementation methods of the present application is that traditional face generation technology is insufficient in fine-grained control of facial expressions.

[0004] To solve the above technical problems, a technical solution adopted in the implementation mode of the present application is: providing a facial video generation model training method, including: obtaining an initial facial image to be processed, initial audio data corresponding to the initial facial image, and an emotion label corresponding to the initial facial image; processing the initial facial image through a preset 3D facial reconstruction tool to obtain first expression feature data; extracting initial audio feature data from the initial audio data, and generating a corresponding emotion one-hot feature vector according to the emotion label; fusing the first expression feature data, the initial audio feature data and the emotion one-hot feature vector to obtain a first facial expression coefficient; injecting preset noise data into the initial facial image to obtain a second facial image; inputting the initial facial image, the second facial image and the first facial expression coefficient into a preset conditional diffusion model to obtain a target facial video with expression.

[0005] Optionally, the initial facial image is input into the 3D face reconstruction tool to obtain corresponding facial geometry data and facial texture data; the facial geometry data and the facial texture data are input into a preset mapping network; the facial geometry data and the facial texture data are subjected to feature space conversion, feature normalization and feature fusion processing through the mapping network to obtain the first expression feature data.

[0006] Optionally, the initial audio feature data is parsed according to a preset attention mechanism to extract corresponding audio key vector data and audio value vector data; the corresponding emotional attention score is calculated according to the audio key vector data and the emotional one-hot feature vector; the audio value vector is weighted according to the emotional attention score to obtain emotional attention feature output data; the first expression feature data and the emotional attention feature output data are input into a preset self-attention deep learning model to obtain the first facial expression coefficient output by the self-attention deep learning model.

[0007] Optionally, the initial face image and the second face image with noise data added are subjected to image stitching processing through the preset conditional diffusion model to obtain an initial video frame; the first face expression coefficient and the initial video frame are input into a preset generator network to obtain continuous video frames of a preset length; and the continuous video frames are combined into the target face video with expression.

[0008] Optionally, the first facial expression coefficient and the initial video frame are input to the encoder module of the U-shaped network structure to obtain input image feature data extracted from the initial video frame, wherein the encoder module includes multiple convolution layers and pooling layers; the input image feature data is processed by the decoder module of the U-shaped network structure to reconstruct image features to obtain continuous video frames of the preset length, wherein the decoder module uses deconvolution or transposed convolution to up-sample the input image feature data.

[0009] Optionally, the first loss value for obtaining the first facial expression coefficient is calculated by presetting a first loss sub-function; the second loss value for obtaining the target facial video with expression is calculated by presetting a second loss sub-function; the training process for obtaining the first facial expression coefficient is optimized according to the first loss value, and the training process for obtaining the target facial video with expression is optimized according to the second loss value, until the first loss sub-function converges and the second loss sub-function converges, to obtain a trained facial video generation model.

[0010] To solve the above technical problems, the second technical solution adopted in the implementation mode of the present application is: to provide a facial video generation method, comprising: obtaining a source image containing a face, source audio data corresponding to the source image, and an emotion setting value corresponding to the source image; inputting the source image, the source audio data and the emotion setting value into a preset facial video generation model to obtain a target facial video with expression, wherein the preset facial video generation model is trained according to the facial video generation model training method as described above.

[0011] To solve the above technical problems, the third technical solution adopted in the implementation mode of the present application is: to provide a facial video generation model training device, including: an initial data acquisition module, used to obtain the initial facial image to be processed, the initial audio data corresponding to the initial facial image, and the emotion label corresponding to the initial facial image; a first data processing module, used to process the initial facial image through a preset 3D facial reconstruction tool to obtain first expression feature data; a second data processing module, used to extract initial audio feature data from the initial audio data, and generate a corresponding emotion single-hot feature vector according to the emotion label; a facial expression coefficient module, used to fuse the first expression feature data, the initial audio feature data and the emotion single-hot feature vector to obtain a first facial expression coefficient; an image noise processing module, used to inject preset noise data into the initial facial image to obtain a second facial image; a facial video generation module, used to input the initial facial image, the second facial image and the first facial expression coefficient into a preset conditional diffusion model to obtain a target facial video with expression.

[0012] To solve the above technical problems, the fourth technical solution adopted in the implementation mode of the present application is: to provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the face video generation model training method as described above, or the face video generation method as described above.

[0013] In order to solve the above-mentioned technical problems, the fifth technical solution adopted in the implementation mode of the present application is: providing a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the face video generation model training method as described above, or the face video generation method as described above.

[0014] Different from the related art, the present application uses a self-attention deep learning model to extract corresponding expression feature coefficients from image data, audio data and emotion label data in advance, and then uses a conditional diffusion model to generate a face video with corresponding expressions based on the extracted expression feature coefficients and image data. This not only improves the performance of generating corresponding face videos based on emotion labels, but also the generated face videos have a finer control granularity of facial expressions and more accurately express the facial expressions corresponding to the emotion labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily described by corresponding drawings, which do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and the figures in the drawings do not constitute proportional limitations unless otherwise stated.

[0016] Figure 1 It is a schematic diagram of the operating environment of the face video generation model training method provided in an embodiment of the present application.

[0017] Figure 2 It is a schematic diagram of the execution flow of the face video generation model training method provided in an embodiment of the present application.

[0018] Figure 3 It is a schematic diagram of the execution flow of obtaining the first expression feature data in the face video generation model training method provided in an embodiment of the present application.

[0019] Figure 4 It is a schematic diagram of the execution flow of obtaining the first facial expression coefficient in the facial video generation model training method provided in an embodiment of the present application.

[0020] Figure 5 It is a schematic diagram of the execution flow of obtaining a target face video with expression in the face video generation model training method provided in an embodiment of the present application.

[0021] Figure 6 It is a schematic diagram of the execution flow of the loss sub-function calculation in the face video generation model training method provided in the embodiment of the present application.

[0022] Figure 7 It is a schematic diagram of the execution flow of the face video generation method provided in an embodiment of the present application.

[0023] Figure 8 It is a system structure diagram of the face video generation model training device provided in an embodiment of the present application.

[0024] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device for executing a face video generation model training method or a face video generation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device schematic diagram or the order in the flow chart.

[0027] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0028] To facilitate understanding of this embodiment, firstly, a face video generation model training method disclosed in the embodiment of the present application is introduced in detail. Figure 1 , Figure 1 Schematic diagram of the operating environment of the face video generation model training method provided in the embodiment of the present application, such as Figure 1 As shown, the execution subject of the face video generation model training method provided in the embodiment of the present application is generally an electronic device with certain computing capabilities, such as a computer device. In some possible implementations, the face video generation model training method can be implemented by a processor calling a computer-readable instruction stored in a memory. Among them, Figure 1 The computer device in the term can be a server. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. It can be understood that Figure 1 The number of computer devices in the figure is only for reference and can be expanded to any number according to actual needs.

[0029] Please continue reading Figure 2 , Figure 2 is a schematic diagram of the execution flow of the face video generation model training method provided in the embodiment of the present application, such as Figure 2 As shown, the process includes the following steps S11 to S16.

[0030] S11, obtaining an initial face image to be processed, initial audio data corresponding to the initial face image, and an emotion label corresponding to the initial face image.

[0031] Among them, the initial face image contains the face image of the target person in the generated face video, the content of the initial audio data is the audio data of the target person speaking in the generated face video, and the emotion label is the emotion expressed by the facial expression and sound when the target person in the generated face video speaks.

[0032] S12: Processing the initial face image by using a preset 3D face reconstruction tool to obtain first expression feature data.

[0033] Among them, 3D facial reconstruction technology is a technology that restores the three-dimensional structure of the human face through image or video data. It can be used to reconstruct a face model with a three-dimensional shape from a two-dimensional image or video, thereby extracting richer and more accurate facial information, expression information, etc. Commonly used 3D face reconstruction tools include OpenFace, Face++, etc.

[0034] As a preferred implementation, please continue to refer to Figure 3 , Figure 3 is a schematic diagram of the execution flow of obtaining the first expression feature data in the face video generation model training method provided in the embodiment of the present application, such as Figure 3 As shown, the following steps S121 to S123 are included.

[0035] S121 . Input an initial face image into a 3D face reconstruction tool to obtain corresponding face geometry data and face texture data.

[0036] Among them, face geometry data describes the 3D shape of the face, including the geometric structure, facial contour, curvature, convexity and concavity of the face. In face geometry data, vertex information is used to describe the position coordinates of each vertex in the face model, through which the 3D shape of the face can be constructed; surface information is used to describe the connection relationship of the face patches or triangles that constitute the face model, and is used to define the geometric structure of the face surface; curvature information is used to describe the curvature, smoothness and other characteristics of the face surface, which is crucial for capturing facial details and realism.

[0037] Among them, the face texture data describes the color, texture, skin details and other information of the face surface, which is used to give the face model visual realism and details. In the face texture data, texture mapping is used to map the color information in the image to the face surface, so that the face model surface presents the real skin color and pattern; skin details include spots, wrinkles, hair and other details, which can be presented on the face model through texture data to increase the realism of the model; lighting information is used to describe the reflection characteristics of the face surface under different lighting conditions, which is crucial for presenting realistic light and shadow effects.

[0038] S122, inputting facial geometry data and facial texture data into a preset mapping network.

[0039] S123. Perform feature space conversion, feature normalization and feature fusion processing on the face geometry data and the face texture data through a mapping network to obtain first expression feature data.

[0040] Among them, different types of face data may be in different feature spaces, and different types of face data need to be converted to the same feature space for subsequent processing. The mapping network maps the face geometry data and face texture data to a common feature space through learning, so that the two have a consistent representation. The converted face data features are normalized, such as mean normalization or standardization, so that the distribution of face data features is more uniform and stable. Feature fusion processing combines the processed face geometry and texture features through weighted summation, splicing or other fusion strategies to form the first expression feature data.

[0041] S13, extracting initial audio feature data from the initial audio data, and generating a corresponding emotion one-hot feature vector according to the emotion label.

[0042] Among them, One-Hot Vector is an encoding method for representing categorical variables, which is usually used in machine learning and deep learning tasks. One-hot vector is a sparse vector in which only one element is 1 and the rest are 0. The only 1 represents the category or label represented by the vector. The corresponding emotion label is represented by a one-hot vector. Each emotion category corresponds to a unique one-hot vector, which makes the representation of the emotion label clear. The one-hot vector represents multiple emotion categories and can cover multiple emotional states at the same time, such as joy, sadness, anger, etc. The one-hot vector form makes the emotion label results output by the model easy to interpret and understand, and can accurately indicate the emotional state of text or voice data.

[0043] S14, fusing the first facial expression feature data, the initial audio feature data and the emotion one-hot feature vector to obtain a first facial expression coefficient.

[0044] As a preferred implementation, please continue to refer to Figure 4 , Figure 4 is a schematic diagram of the execution flow of obtaining the first facial expression coefficient in the facial video generation model training method provided in the embodiment of the present application, such as Figure 4 As shown, the following steps S141 to S144 are included.

[0045] S141. parse the initial audio feature data according to the preset attention mechanism, and extract the corresponding audio key vector data and audio value vector data.

[0046] Among them, in the process of using the attention mechanism to parse the initial audio feature data, the audio special data can be first serialized, and the corresponding dimension alignment and the corresponding position encoding can be added. In the embodiment of the present application, the above-mentioned initial audio feature data is parsed based on the self-attention mechanism, and the above-mentioned audio key vector data represents the vector data of the K (key) type in the self-attention mechanism, the audio value vector data represents the vector data of the V (Value) type in the self-attention mechanism, and the emotion single-hot feature vector represents the vector data of the Q (Query) type in the self-attention mechanism. More technical principles of the self-attention mechanism will not be repeated here.

[0047] S142. Calculate the corresponding emotion attention score according to the audio key vector data and the emotion one-hot feature vector.

[0048] S143. Perform weighted processing on the audio value vector according to the emotional attention score to obtain emotional attention feature output data.

[0049] S144. Input the first facial expression feature data and the emotional attention feature output data into a preset self-attention deep learning model to obtain a first facial expression coefficient output by the self-attention deep learning model.

[0050] Among them, the embodiment of the present application uses a self-attention deep learning model (for example, a Transformer model) to output a first facial expression coefficient. Compared with the traditional attention deep learning model for processing audio data, it can simultaneously capture the connection between global audio data and local audio data, and can more accurately extract emotional features in audio data and reflect the corresponding emotional features on the facial expression coefficient.

[0051] S15. Injecting preset noise data into the initial face image to obtain a second face image.

[0052] S16, inputting the initial face image, the second face image and the first face expression coefficient into a preset conditional diffusion model to obtain a target face video with expression.

[0053] For example, the conditional diffusion model may be a Diffusion model, which gradually generates high-quality images by iteratively introducing noise and gradually reducing the noise. The noise injection, diffusion process (forward diffusion process and reverse diffusion process), gradual image generation, and image generation optimization stages may be performed in sequence.

[0054] As a preferred implementation, please continue to refer to Figure 5 , Figure 5 FIG. 1 is a schematic diagram of an execution flow of obtaining a target face video with expression in the face video generation model training method provided in an embodiment of the present application, such as Figure 5 As shown, it includes the following steps S161 to S163.

[0055] S161. Perform image splicing processing on the initial face image and the second face image with noise data added thereto through a preset conditional diffusion model to obtain an initial video frame.

[0056] S162, inputting the first facial expression coefficient and the initial video frame into a preset generator network to obtain continuous video frames of a preset duration.

[0057] As a preferred implementation, the above step S162 may specifically include: first, input the first facial expression coefficient and the initial video frame to the encoder module of the U-shaped network structure to obtain the input image feature data extracted from the initial video frame, wherein the encoder module includes multiple convolutional layers and pooling layers, the convolutional layer is responsible for extracting the features of the input image, and the pooling layer is used for dimensionality reduction, reducing computational complexity, and retaining the main features, providing a more efficient feature representation for subsequent tasks. Then, the input image feature data is processed by the decoder module of the U-shaped network structure to reconstruct the image features and obtain continuous video frames of a preset length, wherein the decoder module uses deconvolution or transposed convolution to perform upsampling of the input image feature data.

[0058] S163, combining continuous video frames into a target face video with expression.

[0059] As an optional implementation, please continue to refer to Figure 6 , Figure 6 is a schematic diagram of the execution flow of the loss sub-function calculation in the face video generation model training method provided in the embodiment of the present application, such as Figure 6 As shown, it includes the following steps S171 to S173.

[0060] S171. Calculate a first loss value of a first facial expression coefficient by using a preset first loss sub-function.

[0061] S172. Calculate a second loss value of the target facial video with expression by using a preset second loss sub-function.

[0062] S173, optimizing the training process of obtaining the first facial expression coefficient according to the first loss value, and optimizing the training process of obtaining the target facial video with expression according to the second loss value, until the first loss sub-function converges and the second loss sub-function converges, thereby obtaining a trained facial video generation model.

[0063] Please continue reading Figure 7 , Figure 7 FIG. 1 is a schematic diagram of the execution flow of the face video generation method provided in the embodiment of the present application, such as Figure 7As shown, the process includes the following steps S21 to S22.

[0064] S21, obtaining a source image containing a human face, source audio data corresponding to the source image, and an emotion setting value corresponding to the source image.

[0065] S22, inputting source images, source audio data and emotion setting values ​​into a preset face video generation model to obtain a target face video with expression. The preset face video generation model is trained according to the above-mentioned face video generation model training method.

[0066] The facial video generation model training method provided in the embodiment of the present application uses a self-attention deep learning model to extract corresponding expression feature coefficients from image data, audio data and emotion label data in advance, and then uses a conditional diffusion model to generate a facial video with a corresponding expression based on the extracted expression feature coefficients and image data. This not only improves the performance of generating corresponding facial videos based on emotion labels, but also the generated facial videos have a finer control granularity of facial expressions and more accurately express the facial expressions corresponding to the emotion labels.

[0067] Please continue reading Figure 8 , Figure 8 is a schematic diagram of the system structure of the face video generation model training device provided in the embodiment of the present application, such as Figure 8 As shown, the device 80 includes: an initial data acquisition module 81, a first data processing module 82, a second data processing module 83, a facial expression coefficient module 84, an image noise processing module 85 and a facial video generation module 86.

[0068] The initial data acquisition module 81 is used to acquire an initial face image to be processed, initial audio data corresponding to the initial face image, and an emotion label corresponding to the initial face image;

[0069] The first data processing module 82 is used to process the initial face image by using a preset 3D face reconstruction tool to obtain first expression feature data;

[0070] The second data processing module 83 is used to extract initial audio feature data from the initial audio data and generate a corresponding emotion one-hot feature vector according to the emotion label;

[0071] The facial expression coefficient module 84 is used to fuse the first expression feature data, the initial audio feature data and the emotion one-hot feature vector to obtain a first facial expression coefficient;

[0072] The image noise processing module 85 is used to inject preset noise data into the initial face image to obtain a second face image;

[0073] The face video generation module 86 is used to input the initial face image, the second face image and the first face expression coefficient into a preset conditional diffusion model to obtain a target face video with expression.

[0074] As an optional implementation, the first data processing module 82 is specifically used to input the initial facial image into the 3D face reconstruction tool to obtain corresponding facial geometry data and facial texture data; input the facial geometry data and the facial texture data into a preset mapping network; perform feature space conversion, feature normalization and feature fusion processing on the facial geometry data and the facial texture data through the mapping network to obtain the first expression feature data.

[0075] As an optional implementation, the facial expression coefficient module 84 is specifically used to parse the initial audio feature data according to a preset attention mechanism, extract the corresponding audio key vector data and audio value vector data; calculate the corresponding emotional attention score according to the audio key vector data and the emotional one-hot feature vector; perform weighted processing on the audio value vector according to the emotional attention score to obtain emotional attention feature output data; input the first expression feature data and the emotional attention feature output data into a preset self-attention deep learning model to obtain the first facial expression coefficient output by the self-attention deep learning model.

[0076] As an optional implementation, the facial video generation module 86 is specifically used to perform image splicing processing on the initial facial image and the second facial image with noise data added through the preset conditional diffusion model to obtain an initial video frame; input the first facial expression coefficient and the initial video frame into a preset generator network to obtain continuous video frames of a preset length; and combine the continuous video frames into the target facial video with expression.

[0077] As an optional implementation, the facial video generation module 86 is also used to input the first facial expression coefficient and the initial video frame into the encoder module of the U-shaped network structure to obtain input image feature data extracted from the initial video frame, wherein the encoder module includes multiple convolution layers and pooling layers; the input image feature data is processed by the decoder module of the U-shaped network structure to reconstruct image features to obtain continuous video frames of the preset length, wherein the decoder module uses deconvolution or transposed convolution to up-sample the input image feature data.

[0078] As an optional implementation, the facial video generation model training device 80 also includes a loss function calculation module, which is used to calculate the first loss value of the first facial expression coefficient by preset a first loss sub-function; calculate the second loss value of the target facial video with expression by preset a second loss sub-function; optimize the training process of obtaining the first facial expression coefficient according to the first loss value, and optimize the training process of obtaining the target facial video with expression according to the second loss value, until the first loss sub-function converges and the second loss sub-function converges, to obtain a trained facial video generation model with expression.

[0079] It should be noted that the above-mentioned face video generation model training device can execute the face video generation model training method provided in the embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in the embodiment of the face video generation model training device, please refer to the face video generation model training method provided in the embodiment of the present application.

[0080] Please continue reading Fig. 9 , Fig. 9 is a hardware structure diagram of an electronic device 900 for executing a face video generation model training method or a face video generation method provided in an embodiment of the present application, such as Fig. 9 As shown, the electronic device 900 includes:

[0081] One or more processors 910 and memory 920, Fig. 9 A processor 910 is taken as an example.

[0082] The processor 910 and the memory 920 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0083] The memory 920, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the face video generation model training method in the embodiment of the present application, or the program instructions / modules corresponding to the face video generation method. The processor 910 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 920, that is, implementing the face video generation model training method or the face video generation method in the above method embodiment.

[0084] The memory 920 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the face video generation model training device, etc. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 920 may optionally include a memory remotely arranged relative to the processor 910, and these remote memories may be connected to the face video generation model training device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0085] The one or more modules are stored in the memory 920, and when executed by the one or more processors 910, the face video generation model training method or face video generation method in any of the above method embodiments is executed, for example, the above described Figure 2 Steps S11 to S16 of the method, Figure 3 Steps S121 to S123 of the method, Figure 4 Steps S141 to S144 of the method, Figure 5 Steps S161 to S163 of the method, Figure 6 Steps S171 to S173 of the method, Figure 7 Steps S21 to S22 of the method are implemented Figure 8 The functions of modules 81-86 in.

[0086] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0087] The embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, such as Fig. 9 A processor 910 in the embodiment may enable the one or more processors to execute the face video generation model training method or face video generation method in any of the above method embodiments, for example, to execute the above described Figure 2 Steps S11 to S16 of the method, Figure 3 Steps S121 to S123 of the method, Figure 4 Steps S141 to S144 of the method, Figure 5 Steps S161 to S163 of the method, Figure 6Steps S171 to S173 of the method, Figure 7 Steps S21 to S22 of the method are implemented Figure 8 The functions of modules 81-86 in.

[0088] The present application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by the electronic device, the electronic device is enabled to perform the face video generation model training method or face video generation method in any of the above method embodiments, for example, to perform the above described Figure 2 Steps S11 to S16 of the method, Figure 3 Steps S121 to S123 of the method, Figure 4 Steps S141 to S144 of the method, Figure 5 Steps S161 to S163 of the method, Figure 6 Steps S171 to S173 of the method, Figure 7 Steps S21 to S22 of the method are implemented Figure 8 The functions of modules 81-86 in.

[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0090] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; under the idea of ​​the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity; although the present application is described in detail with reference to the above embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application. The non-company software tools or components that appear in the embodiments of the present application are only examples and do not represent actual use.

Claims

1. A facial video generation model training method, characterized in that: include: Acquire an initial face image to be processed, initial audio data corresponding to the initial face image, and an emotion label corresponding to the initial face image; Processing the initial face image by a preset 3D face reconstruction tool to obtain first expression feature data; Extracting initial audio feature data from the initial audio data, and generating a corresponding emotion one-hot feature vector according to the emotion label; fusing the first facial expression feature data, the initial audio feature data and the emotion one-hot feature vector to obtain a first facial expression coefficient; Injecting preset noise data into the initial face image to obtain a second face image; The initial face image, the second face image and the first face expression coefficient are input into a preset conditional diffusion model to obtain a target face video with expression.

2. The facial video generation model training method according to claim 1, characterized in that: The step of processing the initial face image by a preset 3D face reconstruction tool to obtain first expression feature data comprises: Inputting the initial face image into the 3D face reconstruction tool to obtain corresponding face geometry data and face texture data; Inputting the face geometry data and the face texture data into a preset mapping network; The facial geometry data and the facial texture data are subjected to feature space conversion, feature normalization and feature fusion processing through the mapping network to obtain the first expression feature data.

3. The facial video generation model training method according to claim 1, characterized in that: The step of fusing the first expression feature data, the initial audio feature data and the emotion one-hot feature vector to obtain a first facial expression coefficient comprises: Parsing the initial audio feature data according to a preset attention mechanism, extracting corresponding audio key vector data and audio value vector data; Calculate a corresponding emotional attention score according to the audio key vector data and the emotional one-hot feature vector; Performing weighted processing on the audio value vector according to the emotional attention score to obtain emotional attention feature output data; Input the first expression feature data and the emotional attention feature output data into a preset self-attention deep learning model to obtain the first facial expression coefficient output by the self-attention deep learning model.

4. The facial video generation model training method according to claim 1, characterized in that: The step of inputting the initial face image, the second face image and the first face expression coefficient into a preset conditional diffusion model to obtain a target face video with expression includes: Performing image splicing processing on the initial face image and the second face image with noise data added thereto through the preset conditional diffusion model to obtain an initial video frame; Inputting the first facial expression coefficient and the initial video frame into a preset generator network to obtain continuous video frames of a preset duration; The continuous video frames are combined into the target face video with expression.

5. The method for training a facial video generation model according to claim 4, characterized in that: The generator network is a U-shaped network structure, and the step of inputting the first facial expression coefficient and the initial video frame into a preset generator network to obtain continuous video frames of a preset length includes: Inputting the first facial expression coefficient and the initial video frame to the encoder module of the U-shaped network structure to obtain input image feature data extracted from the initial video frame, wherein the encoder module includes a plurality of convolutional layers and pooling layers; The input image feature data is processed by the decoder module of the U-shaped network structure to reconstruct image features and obtain continuous video frames of the preset duration, wherein the decoder module uses deconvolution or transposed convolution to upsample the input image feature data.

6. The method for training a facial video generation model according to claim 1, characterized in that: After the step of inputting the initial face image, the second face image and the first face expression coefficient into a preset conditional diffusion model to obtain a target face video with expression, the method further includes: Calculating the first loss value of the first facial expression coefficient by a preset first loss sub-function; Calculating the second loss value of the target face video with expression by presetting a second loss sub-function; The training process of obtaining the first facial expression coefficient is optimized according to the first loss value, and the training process of obtaining the target facial video with expression is optimized according to the second loss value, until the first loss sub-function converges and the second loss sub-function converges, thereby obtaining a trained facial video generation model.

7. A method for generating a face video, characterized in that: include: Acquire a source image containing a human face, source audio data corresponding to the source image, and an emotion setting value corresponding to the source image; The source image, the source audio data and the emotion setting value are input into a preset face video generation model to obtain a target face video with expression, wherein the preset face video generation model is trained according to the face video generation model training method according to any one of claims 1-6.

8. A facial video generation model training device, characterized in that: include: An initial data acquisition module, used to acquire an initial face image to be processed, initial audio data corresponding to the initial face image, and an emotion label corresponding to the initial face image; A first data processing module, used for processing the initial face image by a preset 3D face reconstruction tool to obtain first expression feature data; A second data processing module, configured to extract initial audio feature data from the initial audio data, and generate a corresponding emotion one-hot feature vector according to the emotion label; A facial expression coefficient module, used for fusing the first expression feature data, the initial audio feature data and the emotion one-hot feature vector to obtain a first facial expression coefficient; An image noise processing module, used for injecting preset noise data into the initial face image to obtain a second face image; The face video generation module is used to input the initial face image, the second face image and the first face expression coefficient into a preset conditional diffusion model to obtain a target face video with expression.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the facial video generation model training method described in any one of claims 1-6, or the facial video generation method described in claim 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the facial video generation model training method described in any one of claims 1 to 6, or the facial video generation method described in claim 7.

Images