Lip shape driven face generation network training method, video generation method and device

Abstract

The present invention relates to a lip shape driven face generation network training method, a video generation method and a device, which are used to solve the problem of generating a real and smooth single object video in real time, comprising: obtaining a training data set, the training data set comprising a complete face image, a masked face image and audio semantic features; constructing a lip shape driven face generation network, comprising a face generator, a face discriminator and a continuous face discriminator; inputting the complete face image, the masked face image and the audio semantic features into the face generator, the face discriminator and the continuous face discriminator for single frame training and multi-frame training, and obtaining the trained face generator, the face discriminator and the continuous face discriminator. The present invention generates a face image by masking a face image, has a strong generalization ability, generates a real, natural and smooth video of a single object, reduces the amount of calculation required for generating a face image by preprocessing the video and reducing the size of the generator model, and improves the real-time performance of the generated video.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and more specifically, to a lip shape driven face generation network training method, a video generation method and a device. Background Art

[0002] With the development of artificial intelligence, digital human generation technology has played an increasingly important role in the fields of film, games and virtual reality, creating an unprecedented immersive experience for users. In this process, real-time lip shape drive, as a key link in the generation of digital humans, achieves real-time synchronization between the lip shape of virtual characters and the audio content when they speak or sing, enhancing the expressiveness and realism of the virtual characters.

[0003] Existing digital human generation algorithms mainly cover three categories: digital human generation based on static images, animated characters, and real videos. First, although digital human videos generated based on static images can drive mouth movements based on audio, they lack natural changes in body movements and facial expressions, resulting in a stiff and incoherent overall effect. This is particularly evident in dynamic or complex scenes, limiting the expressiveness of digital humans in dynamic scenes. Secondly, digital humans generated based on animated characters create animated characters through hand-drawing or three-dimensional modeling, and synchronize audio and mouth movements by controlling facial key points. Although this method has certain flexibility, the generated digital humans often lack realism and are difficult to meet application scenarios with high requirements for immersion and realism. Furthermore, the digital human technology generated based on real videos not only generates lip shapes based on audio features, but also retains the appearance and movement details in the original video, making the generated digital humans more realistic. However, this technology often requires obtaining contextual information of the input video and audio, making it difficult to generate in real time and requiring large amounts of calculation. Therefore, its application is limited in scenarios with high real-time requirements, such as real-time live broadcasts or virtual actor interactions.

[0004] Therefore, how to achieve real-time generation of realistic and smooth digital human videos has become a key research direction. Summary of the invention

[0005] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a lip shape driven face generation network training method, a video generation method and a device for solving the problem of how to generate a realistic and smooth single object video in real time.

[0006] The technical solution adopted by the present invention is a lip shape driven face generation network training method, comprising:

[0007] Acquire a training data set, the training data set comprising a complete face training image of a single object, a masked face training image corresponding to the complete face training image, and training audio semantic features corresponding to the complete face training image;

[0008] Constructing a lip shape driven face generation network, wherein the lip shape driven face generation network includes a face generator, a face discriminator and a continuous face discriminator;

[0009] Performing single-frame training on the face generator and the face discriminator according to the complete face training image, the masked face training image, and the training audio semantic features to obtain a face generator and a face discriminator after single-frame training;

[0010] The face generator and the continuous face discriminator that have been trained by single frame are inputted with the complete face training image, the masked face training image and the training audio semantic features to perform multi-frame training to obtain the face generator and the continuous face discriminator that have been trained by multiple frames.

[0011] Furthermore, the obtaining of the training data set includes:

[0012] Obtaining a training video with audio, wherein the training video corresponds to a single object;

[0013] Obtaining facial key points of each frame of the training video, and cropping each frame of the image based on the facial key points to obtain a complete facial training image;

[0014] Masking the mouth area of ​​each frame of the training video to obtain a masked face training image corresponding to the complete face training image;

[0015] Using a pre-trained semantic feature extraction network to process the audio corresponding to each frame of the training video to obtain a semantic feature of the training audio;

[0016] Constructing training samples, each training sample includes the complete face training image, the masked face training image and the training audio semantic features corresponding to each frame of image;

[0017] The training data set is formed by the training samples.

[0018] Furthermore, the single frame training includes:

[0019] Randomly sampling a first number of training samples from the training data set, and combining all complete facial training images in the first number of training samples to obtain first training reference image features;

[0020] Randomly sampling a training sample from the training data set as a first input sample, inputting the masked face training image and training audio semantic features of the first input sample and the first reference image features into the face generator to obtain a first face generated image;

[0021] Inputting the first generated face image and the complete face training image of the first input sample into the face discriminator respectively, obtaining a first probability value and a second probability value that the first generated face image and the complete face training image of the first input sample are real face images respectively, calculating a first discriminant adversarial loss generated by the face discriminator for the first generated face image and the complete face training image of the first input sample according to the output first probability value and second probability value, updating the parameters of the face discriminator by using the first discriminant adversarial loss, and obtaining an updated face discriminator;

[0022] Predicting the first face generated image using the updated face discriminator to obtain a prediction result, and calculating a first generative adversarial loss according to the prediction result;

[0023] The complete face training image of the first input sample and the first face generated image are respectively input into the pre-trained perceptual network to obtain a first perceptual feature and a second perceptual feature respectively; a first perceptual loss is obtained according to the first perceptual feature and the second perceptual feature; the parameters of the face generator are updated according to the first perceptual loss and the first generative adversarial loss to obtain a face generator after single-frame training update.

[0024] Furthermore, the first generative adversarial loss is calculated according to the prediction result, and the first generative adversarial loss is calculated specifically by the following formula:

[0025]

[0026] Among them, Loss G (sy i ) is the first generation adversarial loss, sy i is the probability value that the prediction result obtained by the updated face discriminator for predicting the i-th first face generated image is a true face image, the MSE function is the mean square error function, i represents the i-th first face generated image input to the face discriminator, and n represents the number of first face generated images input to the face discriminator.

[0027] Further, the acquiring a first perceptual loss according to the first perceptual feature and the second perceptual feature specifically includes:

[0028] The first perceptual loss is calculated by the following formula:

[0029]

[0030] Among them, P real is the first perceptual feature, P fake is the second perceptual feature, Loss P (P real ,P fake ) is the first perceptual loss corresponding to the complete face training image of the first input sample and the first face generated image, L1(P real ,P fake ) is the L1 loss function, j represents the jth layer of the pre-trained perceptual network, and m represents the total number of feature layers of the pre-trained perceptual network.

[0031] Furthermore, the multi-frame training includes:

[0032] randomly sampling a second number of training samples from the training data set, and combining the complete facial training images of the second number of training samples to obtain second reference image features;

[0033] Randomly sampling a third number of continuous training samples from the training data set, and sequentially splicing the masked face training images of the third number of training samples to obtain a spliced ​​masked face training image;

[0034] Inputting the stitched mask face training image, the second reference image features and the training audio semantic features of a third number of continuous training samples into a face generator after single-frame training to obtain a second face generated image;

[0035] Inputting the complete face training images and the second face generated images corresponding to the third continuous number of training samples into the continuous face discriminator respectively, obtaining a third probability value and a fourth probability value that the complete face training images and the second face generated images corresponding to the third continuous number of training samples are continuous face images respectively, calculating a second discriminant adversarial loss generated by the continuous face discriminator for the complete face training images and the second face generated images corresponding to the third continuous number of training samples according to the third probability value and the fourth probability value, and updating the parameters of the continuous face discriminator by using the second discriminant adversarial loss to obtain an updated continuous face discriminator;

[0036] Using the updated continuous face discriminator to predict the second face generated image to obtain a prediction result, and calculating a second generative adversarial loss according to the prediction result;

[0037] Inputting the complete face training image and the second face generation image corresponding to the third number of continuous training samples into the pre-trained perceptual network respectively, and obtaining a third perceptual feature and a fourth perceptual feature respectively; obtaining a second perceptual loss according to the third perceptual feature and the fourth perceptual feature; updating the parameters of the face generator updated after the single-frame training according to the second perceptual loss and the second generation adversarial loss, and obtaining a face generator updated after the multi-frame training;

[0038] The second face generation image and the corresponding training audio semantic features are input into the pre-trained lip synchronization network to obtain a lip synchronization score, the lip synchronization loss is calculated according to the synchronization score, and the parameters of the face generator updated after the multi-frame training are updated according to the lip synchronization loss to obtain the face generator updated after the lip synchronization.

[0039] On the other hand, the present invention provides a method for generating a video by driving a face with a lip shape, comprising the following steps:

[0040] Acquire a target video with audio of a single target object and preprocess the target video to obtain target reference image features and a masked facial target image, facial key points, and a first facial image cropped based on the facial key points;

[0041] Acquire the audio of the target video, and input the audio into the semantic feature extraction network every time the audio of a preset length is obtained to obtain the semantic features of the target audio;

[0042] Inputting the target audio semantic features, the target reference image features and the masked face target image into the face generator to obtain a third face generated image;

[0043] Calculating the pitch angle, roll angle and yaw angle of the face of the first facial image based on the facial key points to obtain the facial orientation of the first facial image;

[0044] Calculating a face fusion mask based on the face orientation of the first face image, and fusing the third face generated image into the image of the target video using the face fusion mask and a Gaussian fusion algorithm to obtain a result generated image;

[0045] The resulting image is used to form a video of the lip-driven face.

[0046] In another aspect, the present invention provides a lip shape driven face generation network training device, the device comprising:

[0047] A training data set acquisition module, wherein the training data set includes a complete face training image of a single object, a masked face training image corresponding to the complete face training image, and training audio semantic features corresponding to the complete face training image;

[0048] A network construction module, used to construct a lip shape driven face generation network, wherein the lip shape driven face generation network includes a face generator, a face discriminator and a continuous face discriminator;

[0049] A single-frame training module, used for performing single-frame training on the face generator and the face discriminator according to the complete face training image, the masked face training image, and the training audio semantic features, to obtain the face generator and the face discriminator after single-frame training;

[0050] The multi-frame training module is used to perform multi-frame training on the face generator and the continuous face discriminator after single-frame training according to the complete face training image, the masked face training image and the training audio semantic feature input to obtain the face generator and the continuous face discriminator after multi-frame training.

[0051] In another aspect, the present invention provides a lip shape driven face video generation device, comprising:

[0052] A target video acquisition and preprocessing module is used to acquire a target video with audio of a single target object and preprocess the target video to obtain target reference image features and a masked facial target image, facial key points, and a first facial image cropped based on the facial key points;

[0053] A target audio semantic feature acquisition module, used to acquire the audio of the target video, and input the audio into the semantic feature extraction network every time the audio of a preset length is acquired, so as to acquire the target audio semantic feature;

[0054] A facial image generation module, configured to input the target audio semantic features, the target reference image features and the masked facial target image into the facial generator to obtain a third facial generated image;

[0055] A face orientation acquisition module, used for calculating the pitch angle, roll angle and yaw angle of the face of the first face image based on the face key points, and acquiring the face orientation of the first face image;

[0056] A result image generation module is used to calculate a face fusion mask based on the face orientation of the first face image, and fuse the third face generated image into the image of the target video using the face fusion mask and the Gaussian fusion algorithm to obtain a result generated image

[0057] The video generation module is used to generate a video of the lip-driven face using the image formed by the result.

[0058] On the other hand, the present invention provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned lip shape driven face generation network training method.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] (1) The lip shape driven face generation network training method provided by the present invention generates a face image by masking the face image, and has a strong generalization ability, so that the generated video of a single object is real, natural and smooth.

[0061] (2) The lip shape driven face generation network training method provided by the present invention greatly reduces the amount of calculation required for the face generator to generate facial images by reducing the model size of the generator part and preprocessing the video, thereby improving the real-time performance of single object video generation.

[0062] (3) The lip shape driven face generation network training method provided by the present invention does not require additional training models to generate the appearance, movements and expressions of a single object video, allowing users to use a face generator to generate a target video through a customized video as needed, thereby meeting different scenarios and needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Schematic diagram of an application scenario of an embodiment of the present invention.

[0064] Figure 2 The figure is a flow chart of a lip shape driven face generation network training method according to an embodiment of the present invention.

[0065] Figure 3 It is a structural schematic diagram of the method for generating video by driving face with lip shape according to an embodiment of the present invention.

[0066] Figure 4 It is a structural schematic diagram of a lip shape driven face generation network training device according to embodiment 3 of the present invention.

[0067] Figure 5 It is a flowchart of the method for generating video by lip shape driven face according to Embodiment 2 of the present invention.

[0068] Figure 6 It is a structural schematic diagram of a lip shape driven face video generation device according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0069] The drawings of the present invention are only for illustrative purposes and should not be construed as limiting the present invention. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0070] Example 1

[0071] For example, Figure 1 A schematic diagram of an application scenario of the lip shape driven face generation network training method provided in an embodiment of the present application, such as Figure 1 As shown, it includes a server 100 and a terminal device 200. The server 100 can be connected by wireless communication or by wired communication.

[0072] It is understandable that the server 100 can be an independent electronic device, a cluster or distributed system composed of multiple electronic devices, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, network services, middleware services, domain name services, big data and artificial intelligence. The operating systems running on the server include but are not limited to Windows, Linux, iOS, Android, etc.

[0073] Figure 1 The number of terminal devices 200 and servers 100 in the figure is merely exemplary, and any number of terminal devices 200 and servers 100 may be included according to actual needs.

[0074] The terminal device 200 includes a smart phone terminal, a personal computer, a tablet computer, a vehicle terminal, an augmented reality or virtual fiber device, a smart wearable device, etc., but is not limited thereto.

[0075] In some embodiments, the server 100 and the terminal device 200 can respectively and separately execute the lip shape driven face generation network training method provided in the embodiment of the present application, or, optionally, the lip shape driven face generation network training method provided in the embodiment of the present application is partially executed in the server 100 and partially executed in the terminal device 200. This embodiment does not limit the specific application environment.

[0076] Figure 2 It is a flowchart of the lip shape driven face generation network training method provided in this embodiment.

[0077] like Figure 2 As shown, the lip shape driven face generation network training method comprises the following steps:

[0078] S1, obtaining a training data set, wherein the training data set includes a complete face training image of a single object, a masked face training image corresponding to the complete face training image, and training audio semantic features corresponding to the complete face training image.

[0079] In this embodiment, obtaining the training data set specifically includes:

[0080] Obtaining a training video with audio, wherein the training video corresponds to a single object;

[0081] Obtaining facial key points of each frame of the training video, and cropping each frame of the image based on the facial key points to obtain a complete facial training image;

[0082] Masking the mouth area of ​​each frame of the training video to obtain a masked face training image corresponding to the complete face training image;

[0083] Using a pre-trained semantic feature extraction network to process the audio corresponding to each frame of the training video to obtain a semantic feature of the training audio;

[0084] Constructing training samples, each training sample includes the complete face training image, the masked face training image and the training audio semantic features corresponding to each frame of image;

[0085] The training data set is formed by the training samples.

[0086] Through the above steps, the entire video is processed to form a training data set including multiple training samples.

[0087] In the specific implementation process, the video with audio can be obtained in a variety of ways, for example, it can be obtained by real-time recording of the subject's speech video, or it can be obtained from a storage medium or from the Internet. This embodiment does not limit the method of obtaining video data. Exemplarily, the video frame rate and audio sampling rate normalization processing of the video data can specifically normalize the video frame rate to 25Fps, and the audio sampling rate to 16000Hz. Every 40ms (corresponding to 640 sampling points) of audio clip length is used as the input of the semantic feature extraction network. After being processed by the semantic feature extraction network, a semantic-level audio feature of 1×768 dimensions is output, and 1 second of audio can generate 25 audio semantic features.

[0088] In this embodiment, the object in the single object may be a real person, a digital person, an animated image, or other objects having lip shape features, which is not limited here.

[0089] In this embodiment, the number of key points for obtaining facial key points can be determined according to actual needs. The more key points there are, the more accurate the facial features can be extracted. However, obtaining too many key points will also increase the complexity of the calculation. For example, in this embodiment, the number of facial key points is 68. Using the key point detection algorithm to crop the face can ensure the integrity of the obtained face and perform a preliminary alignment of the face of the entire video.

[0090] By using the key point detection algorithm to detect each frame of the video, the key point information of the face can be accurately obtained. Using the key points detected to crop the facial image can ensure that the facial area of ​​the cropped facial image remains intact, and at the same time, the facial alignment is preliminarily achieved.

[0091] S2, constructing a lip shape driven face generation network, wherein the lip shape driven face generation network includes a face generator, a face discriminator and a continuous face discriminator.

[0092] The lip shape driven face generation network of this embodiment is a special GAN ​​network (generative adversarial network). Among them, the face generator is used to generate facial images, and the face discriminator is used to discriminate whether the input image is a real face image or a face image generated by the face generator. During training, by alternately training the face generator and the face discriminator, the face generator can generate more and more realistic face images synchronized with the lip shape, so that the face discriminator cannot recognize that it is a generated face image. At the same time, during the training process, the face discriminator continuously improves its recognition ability, thereby improving the ability of the face discriminator to discriminate whether the input image is a real face image or a generated face image. Similarly, by alternately training the face generator and the continuous face discriminator, the face generator can generate a sequence of facial images that is continuous in time and whose lip shape is highly matched with the audio, while the continuous face discriminator is more adept at judging whether multiple frames of facial images remain continuous.

[0093] Through this adversarial training mechanism, the image data distribution generated by the face generator is gradually approached to the real image data distribution, thereby improving the overall performance of the lip shape driven face generation network.

[0094] S3, performing single-frame training on the face generator and the face discriminator according to the complete face training image, the masked face training image, and the training audio semantic features to obtain a face generator and a face discriminator after single-frame training.

[0095] Specifically, single frame training includes the following steps:

[0096] S301, randomly sampling a first number of training samples from the training data set, and combining all complete face training images in the first number of training samples to obtain first training reference image features.

[0097] In a specific implementation process, the first number can be determined according to actual needs, and the first data is greater than or equal to 2. For example, if the first number is five, it is necessary to combine the complete facial images in the five training samples to obtain the first training reference image feature.

[0098] S302, randomly sampling a training sample from the training data set as a first input sample, inputting the masked face training image and training audio semantic features of the first input sample and the first reference image features into the face generator to obtain a first face generated image.

[0099] S303, the first face generation image and the complete face training image of the first input sample are respectively input into the face discriminator, and a first probability value and a second probability value that the first face generation image and the complete face training image of the first input sample are real face images are respectively obtained, and a first discriminant adversarial loss generated by the face discriminator for the first face generation image and the complete face training image of the first input sample is calculated according to the output first probability value and second probability value, and the parameters of the face discriminator are updated by the first discriminant adversarial loss to obtain an updated face discriminator.

[0100] The first discriminative adversarial loss includes the adversarial loss generated by the face discriminator on the complete face training image of the first input sample The adversarial loss generated by the face discriminator on the first face generated image

[0101] The specific calculation formula is as follows:

[0102]

[0103] in, is the adversarial loss generated by the face discriminator for the real face image (i.e., the complete face training image of the first input sample), sx i is a second probability value, i represents the i-th first real face image input to the face discriminator, and n represents the number of real face images input to the face discriminator.

[0104] The specific calculation formula is as follows:

[0105]

[0106] in, is the adversarial loss generated by the face discriminator for the first face generated image, sy i is a first probability value, i represents the i-th first face generated image input to the face discriminator, and n represents the number of first face generated images input to the face discriminator.

[0107] The face discriminator is trained by the above-mentioned adversarial loss function, so that when processing a complete face image, the face discriminator tends to predict the complete face image as 1, and when processing the generated face image, it tends to predict the generated face image as 0. In other words, the face discriminator has the ability to distinguish between real data and generated data.

[0108] Exemplarily, the adaptive learning rate optimization algorithm of this embodiment is implemented using an Adam optimizer, which can adaptively adjust the learning rate of the lip shape driven face generation network and reduce oscillation, thereby improving the training effect and generalization ability.

[0109] S304: Use the updated face discriminator to predict the first face generated image to obtain a prediction result, and calculate a first generative adversarial loss according to the prediction result.

[0110] Specifically, the first generation adversarial loss calculation formula is:

[0111]

[0112] Among them, Loss G (sy i ) is the first generation adversarial loss, sy i is the probability value that the prediction result obtained by the updated face discriminator for predicting the i-th first face generated image is a true face image, the MSE function is the mean square error function, i represents the i-th first face generated image input to the face discriminator, and n represents the number of first face generated images input to the face discriminator.

[0113] In the specific implementation process, based on Loss G (sy i ) The Adam optimizer is used to update the face generator, thereby improving the ability of the face generator to generate face images that make the face discriminator predict a value close to 1, that is, the face generator has the ability to generate face images that tend to be real.

[0114] S304, inputting the complete face training image of the first input sample and the first face generated image into a pre-trained perceptual network respectively, and obtaining a first perceptual feature and a second perceptual feature respectively; obtaining a first perceptual loss according to the first perceptual feature and the second perceptual feature; updating the parameters of the face generator according to the first perceptual loss and the first generative adversarial loss, and obtaining a face generator after single-frame training update.

[0115] The calculation formula of the first perceptual loss is as follows:

[0116]

[0117] Among them, P real is the first perceptual feature, P fake is the second perceptual feature, Loss P (P real ,P fake ) is the first perceptual loss corresponding to the complete face training image of the first input sample and the first face generated image, L1(P real ,P fake ) is the L1 loss function, j represents the jth layer of the pre-trained perceptual network, and m represents the total number of feature layers of the pre-trained perceptual network.

[0118] In the specific implementation process, based on Loss P (P real ,P fake ) uses the Adam optimizer to update the face generator, so that the multi-layer semantic features of the real face image (i.e., the complete face image) and the image generated by the face generator can be made consistent, that is, the real face image and the face generated image can be made similar.

[0119] In this embodiment, the multi-frame training specifically includes the following steps:

[0120] S401, randomly sampling a second number of training samples from the training data set, and combining complete facial training images of the second number of training samples to obtain second reference image features;

[0121] S402, randomly sampling a third number of continuous training samples from the training data set, and sequentially splicing masked face training images of the third number of training samples to obtain spliced ​​masked face training images;

[0122] Exemplarily, in this embodiment, the second number is 5 and the third number is 9, then the complete facial images in the 5 training samples need to be stitched together as the second reference image features. Randomly sample 9 consecutive training samples, stitch the corresponding 9 consecutive masked facial images in the 9 training samples together as the basic input images for multi-frame training, and take the corresponding 9 consecutive audio semantic features and 9 consecutive complete facial images in the 9 training samples as the input of the multi-frame training and the supervision of the generated results, respectively. In the specific implementation process, the specific values ​​of the second number and the third number can be determined according to actual needs, which is only an exemplary description here without specific limitation.

[0123] S403, inputting the stitched mask face training image, the second reference image features, and the training audio semantic features of the third number of continuous training samples into a face generator after single-frame training to obtain a second face generated image;

[0124] S404, inputting the complete face training images corresponding to the third continuous number of training samples and the second face generated image into the continuous face discriminator respectively, obtaining a third probability value and a fourth probability value that the complete face training images corresponding to the third continuous number of training samples and the second face generated image are continuous face images respectively, calculating a second discriminant adversarial loss generated by the continuous face discriminator for the complete face training images corresponding to the third continuous number of training samples and the second face generated image according to the third probability value and the fourth probability value, updating the parameters of the continuous face discriminator by using the second discriminant adversarial loss, and obtaining an updated continuous face discriminator;

[0125] The second discriminative adversarial loss includes the adversarial loss generated by the continuous face discriminator for the complete face training images corresponding to the third number of continuous training samples and the adversarial loss generated by the continuous face discriminator on the second face generated image

[0126] Specifically, the adversarial loss generated by the continuous face discriminator for the complete face training image corresponding to the third number of continuous training samples is The calculation formula is as follows:

[0127]

[0128] Among them, vx i is a third probability value, MSE function is a mean square error function, i represents the complete face training image corresponding to the i-th continuous third number of training samples input into the continuous face discriminator, and n represents the number of complete face training images corresponding to the continuous third number of training samples input into the face discriminator.

[0129] Adversarial loss of the continuous face discriminator on the second face generated image The calculation formula is as follows:

[0130]

[0131] Among them, vy i is a fourth probability value, i represents the i-th second face generated image input to the continuous face discriminator, and n represents the number of the second face generated images input to the face discriminator.

[0132] S405, predicting the second face generated image using the updated continuous face discriminator to obtain a prediction result, and calculating a second generative adversarial loss according to the prediction result;

[0133] In this embodiment, the second generative adversarial loss is calculated using the MSE function, and the calculation formula is:

[0134]

[0135] Among them, Loss GV (vy i ) is the second generation adversarial loss, vy i is the probability of a continuous face image output by the continuous face discriminator for the complete face training images corresponding to the third number of continuous training samples, i represents the i-th second face generated image input to the continuous face discriminator, and n represents the number of the second face generated images input to the continuous face discriminator.

[0136] S406, inputting the complete face training image and the second face generation image corresponding to the third number of continuous training samples into the pre-trained perceptual network respectively, and obtaining a third perceptual feature and a fourth perceptual feature respectively; obtaining a second perceptual loss according to the third perceptual feature and the fourth perceptual feature; updating the parameters of the face generator updated after the single-frame training according to the second perceptual loss and the second generative adversarial loss, and obtaining a face generator updated after the multi-frame training;

[0137] In the specific implementation process, the Adam optimizer can be used based on Loss GV (vy i ) Update the face generator. Using the Adam optimizer to update the face generator can make the face generator generate multiple frames of face images that make the continuous face discriminator predict tend to 1, that is, the face generator can generate images that tend to be continuous real faces. The continuous face discriminator introduces implicit supervision to the lip shape driven face generation network, making the generated multiple frames of face images more continuous, without mutations that affect the authenticity of the subsequent generated video.

[0138] S408, inputting the second facial generated image and the corresponding training audio semantic features into the pre-trained lip synchronization network to obtain a lip synchronization score, calculating the lip synchronization loss according to the synchronization score, updating the parameters of the face generator updated after the multi-frame training according to the lip synchronization loss, and obtaining the face generator updated after the lip synchronization.

[0139] In the specific implementation process, the lip synchronization network can be implemented using the Syncnet synchronization network. In this embodiment, the lip synchronization loss is calculated using the mean square error loss function MSE, and the specific calculation formula is as follows:

[0140]

[0141] Among them, Loss GS (my i ) is the lip sync loss, my iis a lip synchronization score, i represents the i-th second face generated image input into the lip synchronization network, and n represents the number of second face generated images input into the lip synchronization network.

[0142] S4, performing multi-frame training on the face generator and the continuous face discriminator after single-frame training according to the complete face training image, the masked face training image and the training audio semantic feature input, to obtain the multi-frame trained face generator and the continuous face discriminator.

[0143] Example 2

[0144] The present application also discloses a method for generating a video by driving a face with a lip shape, such as Figure 3 and Figure 5 As shown, including:

[0145] S51, obtaining a target video with audio of a single target object and preprocessing the target video to obtain target reference image features and a masked facial target image, facial key points, and a first facial image cropped based on the facial key points;

[0146] S52, obtaining the audio of the target video, and inputting the audio into the semantic feature extraction network every time the audio of a preset length is obtained to obtain the semantic features of the target audio;

[0147] S53, inputting the target audio semantic feature, the target reference image feature and the masked face target image into the face generator to obtain a third face generated image;

[0148] S54, calculating the pitch angle, roll angle and yaw angle of the face of the first facial image based on the facial key points, and acquiring the facial orientation of the first facial image;

[0149] S55, calculating a face fusion mask based on the face orientation of the first face image, and fusing the third face generated image into the image of the target video using the face fusion mask and a Gaussian fusion algorithm to obtain a result generated image;

[0150] S56, generating an image through the result to form a video of the lip-driven face.

[0151] In this embodiment, step S51 specifically includes the following steps:

[0152] S511, a target video with audio of a single target object is used as an original video, a reverse video of the original video is obtained, and the original video and the reverse video are connected end to end to obtain a loop video;

[0153] S512, obtaining facial key points of each frame of the original video, and filtering the facial key points between each frame of the image using a filter to obtain filtered facial key points;

[0154] In a specific implementation process, the number of facial key points can be determined according to actual application scenarios. For example, the number of facial key points in this embodiment is 68. In this embodiment, the filter is a Euro key point filter.

[0155] S513, cropping each frame of the image based on the filtered facial key points to obtain a first facial image;

[0156] S514, randomly acquiring a plurality of frames of first facial images for splicing to obtain target reference image features;

[0157] S515: Mask the mouth area of ​​the first facial image according to the filtered facial key points to obtain a masked facial target image.

[0158] Example 3

[0159] This embodiment provides a lip shape driven face generation network training device 300, such as Figure 4 As shown, it includes a training data set acquisition module 301, a network construction module 302, a single frame training module 303, and a multi-frame training module 304. Among them:

[0160] The training data set acquisition module 301 can be used to perform Figure 2 As shown in step S1, for the detailed description of the training data set acquisition module 301, reference may be made to the description of step S1.

[0161] The network construction module 302 can be used to perform Figure 2 As shown in step S2, for the detailed description of the training data set acquisition module 302, reference may be made to the description of step S2.

[0162] The single frame training module 303 can be used to perform Figure 2 As shown in step S3, for the specific description of the single frame training module 303, reference may be made to the description of step S3.

[0163] The multi-frame training module 304 can be used to perform Figure 2 As shown in step S4, for the detailed description of the multi-frame training module 304, reference may be made to the description of step S4.

[0164] It can be understood that the above-mentioned device embodiment and the above-mentioned method embodiment can correspond to each other, and similar descriptions of the device embodiment can refer to the method embodiment. To avoid repetition, it will not be repeated here. A lip-shaped driven face generation network training device provided in an embodiment of the present application can execute a lip-shaped driven face generation network training method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the lip-shaped driven face generation network training device can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules.

[0165] Example 4

[0166] This embodiment provides a lip shape driven face video generation device 400, such as Figure 6 As shown, it includes a target video acquisition and preprocessing module 401, a target audio semantic feature acquisition module 402, a face image generation module 403, a face orientation acquisition module 404, a result image generation module 405, and a video generation module 406. Among them:

[0167] The target video acquisition and preprocessing module 401 can be used to perform Figure 5 As shown in step S51 , for a detailed description of the target video acquisition and preprocessing module 401 , reference may be made to the description of step S51 .

[0168] The target audio semantic feature acquisition module 402 can be used to perform Figure 5 As shown in step S52, for the specific description of the target audio semantic feature acquisition module 402, reference may be made to the description of step S52.

[0169] The facial image generation module 403 can be used to perform Figure 5 As shown in step S53, for the detailed description of the facial image generating module 403, reference may be made to the description of step S53.

[0170] The face orientation acquisition module 404 may be used to perform Figure 5 As shown in step S54, for the detailed description of the face orientation acquisition module 404, reference may be made to the description of step S54.

[0171] The result image generation module 405 can be used to perform Figure 5 As shown in step S55, for the specific description of the result image generating module 405, reference may be made to the description of step S55.

[0172] The video generation module 406 can be used to perform Figure 5 As shown in step S56, for the detailed description of the video generation module 406, reference may be made to the description of step S56.

[0173] It can be understood that the above-mentioned device embodiment and the above-mentioned method embodiment can correspond to each other, and similar descriptions of the device embodiment can refer to the method embodiment. To avoid repetition, it will not be repeated here. A lip-shaped driven face video generation device provided in an embodiment of the present application can execute a lip-shaped driven face video generation method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the lip-shaped driven face video generation device can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules.

[0174] Example 5

[0175] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed, a lip shape driven face generation network training method as in Embodiment 1 is implemented.

[0176] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0177] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A lip shape driven face generation network training method, characterized in that: include: Acquire a training data set, the training data set comprising a complete face training image of a single object, a masked face training image corresponding to the complete face training image, and training audio semantic features corresponding to the complete face training image; Constructing a lip shape driven face generation network, wherein the lip shape driven face generation network includes a face generator, a face discriminator and a continuous face discriminator; Performing single-frame training on the face generator and the face discriminator according to the complete face training image, the masked face training image, and the training audio semantic features to obtain a face generator and a face discriminator after single-frame training; According to the complete face training image, the masked face training image and the training audio semantic features, the face generator and the continuous face discriminator after single-frame training are input for multi-frame training to obtain the face generator and the continuous face discriminator after multi-frame training; The single frame training comprises: Randomly sampling a first number of training samples from the training data set, and combining all complete facial training images in the first number of training samples to obtain first training reference image features; Randomly sampling a training sample from the training data set as a first input sample, inputting the masked face training image and training audio semantic features of the first input sample and the first training reference image features into the face generator to obtain a first face generated image; Inputting the first generated face image and the complete face training image of the first input sample into the face discriminator respectively, obtaining a first probability value and a second probability value that the first generated face image and the complete face training image of the first input sample are real face images respectively, calculating a first discriminant adversarial loss generated by the face discriminator for the first generated face image and the complete face training image of the first input sample according to the output first probability value and second probability value, updating the parameters of the face discriminator by using the first discriminant adversarial loss, and obtaining an updated face discriminator; Predicting the first face generated image using the updated face discriminator to obtain a prediction result, and calculating a first generative adversarial loss according to the prediction result; Inputting the complete face training image of the first input sample and the first face generation image into the pre-trained perceptual network respectively, and obtaining a first perceptual feature and a second perceptual feature respectively; obtaining a first perceptual loss according to the first perceptual feature and the second perceptual feature; updating the parameters of the face generator according to the first perceptual loss and the first generative adversarial loss, and obtaining a face generator after single-frame training update; The first generative adversarial loss is calculated according to the prediction result, and the first generative adversarial loss is calculated specifically by the following formula: in, is the first generation adversarial loss, is the updated face discriminator for the first The probability value of the predicted result obtained by predicting the first face generated image is the real face image, The function is the mean square error function, represents the first The first face generated image, Indicates the number of first face generation images input to the face discriminator.

2. A lip shape driven face generation network training method according to claim 1, characterized in that: The obtaining of the training data set comprises: Obtaining a training video with audio, wherein the training video corresponds to a single object; Obtaining facial key points of each frame of the training video, and cropping each frame of the image based on the facial key points to obtain a complete facial training image; Masking the mouth area of ​​each frame of the training video to obtain a masked face training image corresponding to the complete face training image; Using a pre-trained semantic feature extraction network to process the audio corresponding to each frame of the training video to obtain a semantic feature of the training audio; Constructing training samples, each training sample includes the complete face training image, the masked face training image and the training audio semantic features corresponding to each frame of image; The training data set is formed by the training samples.

3. The lip shape driven face generation network training method according to claim 1, characterized in that: The obtaining a first perceptual loss according to the first perceptual feature and the second perceptual feature specifically includes: The first perceptual loss is calculated by the following formula: in, is the first perceptual feature, is the second perceptual feature, A first perceptual loss corresponding to the complete face training image of the first input sample and the first face generated image, for Loss function, Represents the pre-trained perception network layer, Represents the total number of feature layers of the pre-trained perception network.

4. A lip shape driven face generation network training method according to any one of claims 2-3, characterized in that: The multi-frame training comprises: randomly sampling a second number of training samples from the training data set, and combining the complete facial training images of the second number of training samples to obtain second reference image features; Randomly sampling a third number of continuous training samples from the training data set, and sequentially splicing the masked face training images of the third number of training samples to obtain a spliced ​​masked face training image; Inputting the stitched mask face training image, the second reference image features and the training audio semantic features of a third number of continuous training samples into a face generator after single-frame training to obtain a second face generated image; Inputting the complete face training images and the second face generated images corresponding to the third continuous number of training samples into the continuous face discriminator respectively, obtaining a third probability value and a fourth probability value that the complete face training images and the second face generated images corresponding to the third continuous number of training samples are continuous face images respectively, calculating a second discriminant adversarial loss generated by the continuous face discriminator for the complete face training images and the second face generated images corresponding to the third continuous number of training samples according to the third probability value and the fourth probability value, and updating the parameters of the continuous face discriminator by using the second discriminant adversarial loss to obtain an updated continuous face discriminator; Using the updated continuous face discriminator to predict the second face generated image to obtain a prediction result, and calculating a second generative adversarial loss according to the prediction result; Inputting the complete face training image and the second face generation image corresponding to the third number of continuous training samples into the pre-trained perceptual network respectively, and obtaining a third perceptual feature and a fourth perceptual feature respectively; obtaining a second perceptual loss according to the third perceptual feature and the fourth perceptual feature; updating the parameters of the face generator updated after the single-frame training according to the second perceptual loss and the second generation adversarial loss, and obtaining a face generator updated after the multi-frame training; The second face generation image and the corresponding training audio semantic features are input into the pre-trained lip synchronization network to obtain a lip synchronization score, the lip synchronization loss is calculated according to the synchronization score, and the parameters of the face generator updated after the multi-frame training are updated according to the lip synchronization loss to obtain the face generator updated after the lip synchronization.

5. A method for generating a video by driving a face with a lip shape, characterized in that: The lip shape driven face generation network trained by the lip shape driven face generation network training method according to any one of claims 1 to 4 comprises the following steps: Acquire a target video with audio of a single target object and preprocess the target video to obtain target reference image features and a masked facial target image, facial key points, and a first facial image cropped based on the facial key points; Acquire the audio of the target video, and input the audio into a semantic feature extraction network every time a preset length of audio is obtained to obtain a semantic feature of the target audio; Inputting the target audio semantic features, the target reference image features and the masked face target image into the face generator to obtain a third face generated image; Calculating the pitch angle, roll angle and yaw angle of the face of the first facial image based on the facial key points to obtain the facial orientation of the first facial image; Calculating a face fusion mask based on the face orientation of the first face image, and fusing the third face generated image into the image of the target video using the face fusion mask and a Gaussian fusion algorithm to obtain a result generated image; The resulting image is used to form a video of the lip-driven face.

6. A lip shape driven face generation network training device, characterized in that: include: A training data set acquisition module, wherein the training data set includes a complete face training image of a single object, a masked face training image corresponding to the complete face training image, and training audio semantic features corresponding to the complete face training image; A network construction module, used to construct a lip shape driven face generation network, wherein the lip shape driven face generation network includes a face generator, a face discriminator and a continuous face discriminator; A single-frame training module, used for performing single-frame training on the face generator and the face discriminator according to the complete face training image, the masked face training image, and the training audio semantic features, to obtain the face generator and the face discriminator after single-frame training; A multi-frame training module, used for inputting the face generator and the continuous face discriminator after single-frame training to perform multi-frame training according to the complete face training image, the masked face training image and the training audio semantic features, so as to obtain the face generator and the continuous face discriminator after multi-frame training; The single frame training comprises: Randomly sampling a first number of training samples from the training data set, and combining all complete facial training images in the first number of training samples to obtain first training reference image features; Randomly sampling a training sample from the training data set as a first input sample, inputting the masked face training image and training audio semantic features of the first input sample and the first training reference image features into the face generator to obtain a first face generated image; Inputting the first generated face image and the complete face training image of the first input sample into the face discriminator respectively, obtaining a first probability value and a second probability value that the first generated face image and the complete face training image of the first input sample are real face images respectively, calculating a first discriminant adversarial loss generated by the face discriminator for the first generated face image and the complete face training image of the first input sample according to the output first probability value and second probability value, updating the parameters of the face discriminator by using the first discriminant adversarial loss, and obtaining an updated face discriminator; Predicting the first face generated image using the updated face discriminator to obtain a prediction result, and calculating a first generative adversarial loss according to the prediction result; Inputting the complete face training image of the first input sample and the first face generation image into the pre-trained perceptual network respectively, and obtaining a first perceptual feature and a second perceptual feature respectively; obtaining a first perceptual loss according to the first perceptual feature and the second perceptual feature; updating the parameters of the face generator according to the first perceptual loss and the first generative adversarial loss, and obtaining a face generator after single-frame training update; The first generative adversarial loss is calculated according to the prediction result, and the first generative adversarial loss is calculated specifically by the following formula: in, is the first generation adversarial loss, is the updated face discriminator for the first The probability value of the predicted result obtained by predicting the first face generated image is the real face image, The function is the mean square error function, represents the first The first face generated image, Indicates the number of first face generation images input to the face discriminator.

7. The lip shape driven face generation network training device according to claim 6, characterized in that: The method for obtaining the training data set includes: Obtaining a training video with audio, wherein the training video corresponds to a single object; Obtaining facial key points of each frame of the training video, and cropping each frame of the image based on the facial key points to obtain a complete facial training image; Masking the mouth area of ​​each frame of the training video to obtain a masked face training image corresponding to the complete face training image; Using a pre-trained semantic feature extraction network to process the audio corresponding to each frame of the training video to obtain a semantic feature of the training audio; Constructing training samples, each training sample includes the complete face training image, the masked face training image and the training audio semantic features corresponding to each frame of image; The training data set is formed by the training samples.

8. The lip shape driven face generation network training device according to claim 6, characterized in that: The obtaining a first perceptual loss according to the first perceptual feature and the second perceptual feature specifically includes: The first perceptual loss is calculated by the following formula: in, is the first perceptual feature, is the second perceptual feature, A first perceptual loss corresponding to the complete face training image of the first input sample and the first face generated image, for Loss function, Represents the pre-trained perception network layer, Represents the total number of feature layers of the pre-trained perception network.

9. A lip shape driven face video generation device, characterized in that: The lip shape driven face generation network trained by the lip shape driven face generation network training device according to any one of claims 6 to 8 comprises: A target video acquisition and preprocessing module is used to acquire a target video with audio of a single target object and preprocess the target video to obtain target reference image features and a masked facial target image, facial key points, and a first facial image cropped based on facial key points; A target audio semantic feature acquisition module is used to acquire the audio of the target video, and input the audio into a semantic feature extraction network every time a preset length of audio is acquired to acquire the target audio semantic feature; A facial image generation module, configured to input the target audio semantic features, the target reference image features and the masked facial target image into the facial generator to obtain a third facial generated image; A face orientation acquisition module, used for calculating the pitch angle, roll angle and yaw angle of the face of the first face image based on the face key points, and acquiring the face orientation of the first face image; A result image generation module is used to calculate a face fusion mask based on the face orientation of the first face image, and fuse the third face generated image into the image of the target video using the face fusion mask and the Gaussian fusion algorithm to obtain a result generated image The video generation module is used to generate a video of the lip-driven face using the image formed by the result.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lip shape driven face generation network training method according to any one of claims 1 to 4 is implemented.

Images