Face video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a face video generation method and device, electronic equipment and a storage medium, and belongs to the technical field of deep learning. The method comprises the following steps: acquiring a target voice for driving a target object to give out; performing feature extraction on the target voice to obtain voice features; inputting the voice features into a face key point prediction module for key point generation to obtain a plurality of reference key point data; acquiring first face key point data and second face key point data of the target object; performing data adjustment on the plurality of reference key point data based on the first face key point data to obtain target face key point data; and performing face video generation based on the target face key point data and the second face key point data to obtain a target face video. According to the embodiment of the invention, under the condition that the duration of the video data provided by the user is relatively short, the accuracy of the generated target face video can be ensured, and meanwhile, the efficiency of generating the target face video is improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method and device for generating a face video, an electronic device, and a storage medium. Background Art

[0002] 2D digital human is a digital virtual character presented in the form of flat graphics. 2D digital human can create a virtual image for users, solving the problem of real people appearing on camera, and is widely used in film and television production, online live broadcast and other fields.

[0003] The 2D digital human generation model can be driven by text or voice based on a single image or a video of the object provided by the object, and generate a spoken video corresponding to the object's face. When the related technology generates a face video based on audio drive, the video data is input into the 2D digital human generation model to generate a face video according to a video data provided by the object. This method requires the object to provide a long period of video data to ensure the accuracy of the mouth shape in the generated face video, resulting in low efficiency in generating face videos. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a facial video generation method and device, an electronic device and a storage medium, which can ensure the accuracy of the generated target facial video even when the video data provided by the user is short in length, while improving the efficiency of generating the target facial video.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for generating a face video, the method comprising:

[0006] Acquire a target voice for driving a target object to utter, the target voice comprising a plurality of continuous time zones, each of the time zones corresponding to a frame in a target face video of the target object;

[0007] Extracting features of the target speech to obtain speech features;

[0008] Inputting the speech feature into a pre-trained facial key point prediction module to generate key points, and obtaining a plurality of reference key point data, each of which corresponds to the speech feature of a time zone of the target speech;

[0009] Acquire first facial key point data and second facial key point data of the target object, wherein the first facial key point data is used to represent key point data that is the same as the reference key point data in the facial region of the target object, and the second facial key point data is used to represent key point data that is different from the reference key point data in the facial region of the target object;

[0010] Performing data adjustment on the plurality of reference key point data based on the first facial key point data to obtain target facial key point data;

[0011] A facial video is generated based on the target facial key point data and the second facial key point data to obtain the target facial video.

[0012] In some embodiments, obtaining the first facial key point data and the second facial key point data of the target object includes:

[0013] Acquire an object video of the target object;

[0014] Acquiring a reference video frame from the object video;

[0015] Perform image extraction based on the reference video frame to obtain a facial image of the object;

[0016] Performing image segmentation on the facial image of the target object to obtain a first facial image and a second facial image, wherein the first facial image represents a facial image corresponding to a mouth area of ​​the target object, and the second facial image represents a facial image outside the mouth area of ​​the target object;

[0017] Extracting key points from the first facial image to obtain first facial key point data;

[0018] Key point extraction is performed on the second facial image to obtain second facial key point data.

[0019] In some embodiments, the acquiring of reference video frames from the object video comprises:

[0020] Obtaining the shooting clarity and shooting completeness of each video frame in the object video;

[0021] Comparing the shooting clarity in each of the video frames to obtain a shooting clarity comparison result;

[0022] Comparing the shooting completeness in each of the video frames to obtain a shooting completeness comparison result;

[0023] Based on the shooting clarity comparison result and the shooting completeness comparison result, a reference video frame is determined in the object video.

[0024] In some embodiments, the facial key point prediction module includes a feature fusion module and a decoding module, and the speech feature is input into the pre-trained facial key point prediction module to generate key points to obtain a plurality of reference key point data, including:

[0025] Sample random vectors from a preset standard normal distribution space;

[0026] By means of the feature fusion module, the random vector and the speech feature are subjected to feature fusion to obtain a feature fusion vector;

[0027] The feature fusion vector is input into the decoding module for decoding to obtain a plurality of reference key point data.

[0028] In some embodiments, the step of adjusting the plurality of reference key point data based on the first facial key point data to obtain target facial key point data includes:

[0029] Determine a first facial key point data mean and a first facial key point data variance according to the first facial key point data;

[0030] Determine a mean value of the reference key point data and a variance of the reference key point data according to the plurality of reference key point data;

[0031] Based on the first facial key point data mean, the first facial key point data variance, the reference key point data mean and the reference key point data variance, each of the reference key point data is normalized to obtain target facial key point data.

[0032] In some embodiments, generating a face video based on the target face key point data and the second face key point data to obtain the target face video includes:

[0033] Encoding the reference video frame and the target facial key point data by a preset encoder to obtain a target feature vector and a target offset vector;

[0034] By using a preset face image generator, an image is generated based on the target feature vector and the target bias vector to obtain a target mouth shape adjustment image;

[0035] Generate a target facial image excluding a mouth area of ​​the target object according to the second facial key point data;

[0036] The target mouth shape adjustment image and the target facial image are subjected to video synthesis to generate the target face video.

[0037] In some embodiments, generating a target facial image excluding a mouth area of ​​the target object according to the second facial key point data includes:

[0038] Extracting emotional features from the target speech to obtain emotional features;

[0039] Classifying the emotional features based on a preset emotional classification model to obtain the emotional state of the target object;

[0040] Adjusting the key point positions of the second facial key point data according to the emotional state of the target object to generate facial key point data corresponding to the emotional state of the target object;

[0041] An image is generated based on the facial key point data corresponding to the emotional state of the target object to obtain an image of the target facial area excluding the mouth area of ​​the object.

[0042] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a face video generation device.

[0043] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0045] The face video generation method and device, electronic device and storage medium proposed in the present application obtain a target voice for driving the target object to emit; extract the features of the target voice to obtain voice features; input the voice features into the facial key point prediction module to generate key points, and obtain multiple reference key point data, so that the facial representation of the target object can be predicted according to the target voice. Then, based on the video data provided by the user, the first facial key point data and the second facial key point data of the target object can be respectively obtained; based on the first facial key point data, the multiple reference key point data are adjusted to obtain the target facial key point data. Since the first facial key point data represents the key point data that is the same as the multiple reference key point data in the facial area of ​​the target object, the predicted multiple reference data key points can be adjusted based on the first facial key point data of the target object, so as to ensure the accuracy of the generated target face video. Furthermore, the face video can be generated based on the target facial key point data and the second facial key point data to obtain the target face video. Since only the reference key point data needs to be fine-tuned according to the first facial key point data, the time of face video generation is reduced, and the efficiency of face video generation is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a method for generating a face video provided in an embodiment of the present application;

[0047] Figure 2is a schematic diagram of a face video generation model provided in an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of a speech feature extraction module provided in an embodiment of the present application;

[0049] Figure 4 is a schematic diagram of a facial key point prediction module provided in an embodiment of the present application;

[0050] Figure 5 Schematic diagram of the training process of the facial key point prediction module provided in the embodiment of the present application;

[0051] Figure 6 is a schematic diagram of a face generation module provided in an embodiment of the present application;

[0052] Figure 7 is a schematic diagram of the first training stage of the face generation module provided in an embodiment of the present application;

[0053] Figure 8 is a schematic diagram of the second training stage of the face generation module provided in an embodiment of the present application;

[0054] Fig. 9 It is a structural schematic diagram of a face video generating device provided in an embodiment of the present application;

[0055] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] First, some nouns involved in this application are analyzed:

[0060] Digital human: Digital human is a virtual character image close to human image created by digital technology, such as some virtual internet celebrities, virtual singers, virtual actors, or virtual spokespersons of enterprises, which can replace real people to record videos and live broadcasts.

[0061] 3D face morphable statistical model (3DMM): 3DMM is a parametric statistical model of a three-dimensional face that can represent changes in face shape and appearance based on a statistical model. It is generally represented by key point position parameters. 3DMM is commonly used in computer vision, image processing, animation production and other fields.

[0062] Before introducing the facial video generation method provided by the embodiment of the present application, the related technology is first introduced. In the related technology, when customizing a facial video for a certain user, the user is required to provide a relatively long video data to train the facial video generation model to ensure that the mouth shape in the generated facial video is as close as possible to the user's actual mouth shape when speaking. The length of the video data generally ranges from 10 to 20 minutes, and the requirements for the video data provided by the user are relatively high, which makes the generation of facial videos inefficient. In addition, when generating facial videos, the related technology cannot avoid the influence of the human body or facial posture on the predicted key points of the mouth, resulting in low accuracy of the predicted mouth key points, and cannot be personalized with the mouth shapes of different users when speaking, that is, the predicted mouth shape is inaccurate.

[0063] Based on this, the embodiments of the present application provide a facial video generation method and device, an electronic device and a storage medium, which aim to improve the accuracy and efficiency of generating facial videos while using shorter video data provided by users.

[0064] The facial video generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the facial video generation method in the embodiments of the present application is described.

[0065] The facial video generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The facial video generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the facial video generation method, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] Figure 1 is an optional flowchart of the face video generation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps 101 to 106:

[0068] Step 101, obtaining a target voice for driving a target object to speak;

[0069] Step 102, extracting features of the target speech to obtain speech features;

[0070] Step 103, inputting the facial key point prediction module speech features into a pre-trained facial key point prediction module to generate key points, and obtaining a plurality of reference key point data, each facial key point prediction module reference key point data corresponding to the facial key point prediction module speech features of a time zone of the facial key point prediction module target speech;

[0071] Step 104, obtaining first facial key point data and second facial key point data of a target object of a facial key point prediction module;

[0072] Step 105, adjusting the reference key point data of multiple facial key point prediction modules based on the first facial key point data of the facial key point prediction module to obtain target facial key point data;

[0073] Step 106, generating a face video based on the target face key point data of the face key point prediction module and the second face key point data of the face key point prediction module to obtain the target face video of the face key point prediction module.

[0074] In step 101 of some embodiments, the target object is a virtual image in a video generated according to the face of the target user, and is an object that is lip-driven to emit a target voice. The target voice is a voice used to drive the target object to emit. The target voice can be a broadcast voice for news broadcasting, or a song recording, etc. The target voice is expressed as a voice waveform that changes in amplitude over time. Among them, the target voice includes multiple continuous time zones, and each time zone represents a voice segment divided from the target voice according to a preset voice time length, that is, a voice waveform segment. The preset voice time length can be equal or unequal, for example, it can be one second, and each time zone corresponds to a frame in the target face video of the target object. After obtaining the facial key point information of the target object corresponding to the time zone, the target object can be lip-driven in the video frame corresponding to the time zone according to the facial key points corresponding to each time zone while playing the target voice according to the time zone. The target can obtain the target voice by downloading it from the Internet, obtaining it from some public voice databases, or directly provided by the target user, which is not limited here.

[0075] Please refer to Figure 2 , Figure 2: is a schematic diagram of a face video generation model provided by an embodiment of the present application. Among them, the face video generation model 200 may include a speech feature extraction module 210, a facial key point prediction module 220, and a face generation module 230. Specifically, the speech feature extraction module 210 is used to extract the speech features of the target speech, and input the extracted speech features into the facial key point prediction module 220. The speech feature extraction module 210 provided in the embodiment of the present application is obtained by collecting multiple hours of Chinese and English videos for training, wherein the collected multiple hours of Chinese and English video data are screened, and the influence of factors such as multiple voices (sounds other than the voice of the target object), background sounds and noise are removed to ensure the clarity of the audio of the target speaker in the training video data; the facial key point prediction module 220 is used to predict the corresponding facial key points according to the speech features; the face generation module 230 is used to generate the corresponding target face video according to the facial key points obtained by the facial key point prediction module 220.

[0076] In step 102 of some embodiments, specifically, the speech features of the target speech at 16 kilohertz (KHz) are extracted by the speech feature extraction module 210. The speech features of 16 KHz have excellent resistance to interference from background sounds and noises, and can avoid interference from noises on the target speech, thereby improving the accuracy of the extracted speech features.

[0077] See also Figure 3 , Figure 3 Schematic diagram of the speech feature extraction module 210 provided in the embodiment of the present application. The speech feature extraction module 210 includes a convolution feature extraction submodule 211 and a conversion submodule 212. The convolution feature extraction submodule 211 is used to extract local features of the input target speech, and the conversion submodule 212 is used to extract global features of the target speech. The conversion submodule 212 can be a transformer model, which includes many conversion layers, and each conversion layer can be a transformer block.

[0078] In this embodiment, the convolution feature extraction submodule 211 can be composed of a module composed of multiple one-dimensional convolution layers and Gaussian error linear unit activation layers in series, and there is no limit on the number of modules composed of one-dimensional convolution layers and Gaussian error linear unit activation layers in series. Among them, the activation function used by the Gaussian error linear unit activation layer is the Gaussian Error Linear Unit (Gaussian Error Linear Unit, GeLU) activation function. The conversion submodule 212 is composed of multiple transformer blocks, and each transformer block is composed of an attention mechanism, a dropout layer and a normalization layer. Among them, the attention mechanism can make the speech feature extraction module 210 focus on specific speech features when performing speech feature extraction to prevent interference from other speech features; the dropout layer can temporarily make the output of certain nodes in certain neural network models temporarily zero to prevent the occurrence of overfitting; the normalization layer is used to standardize the input data to reduce the occurrence of gradient explosion to improve the stability of the training process.

[0079] Furthermore, the speech features extracted by the speech feature extraction module 210 may include the pitch, timbre, speed, speech rhythm and speech intensity of the target speech, and the extracted speech features are input into the facial key point prediction module 220 for facial key point prediction.

[0080] In step 103 of some embodiments, the speech features are input into a pre-trained facial key point prediction module to generate key points, and a plurality of reference key point data are obtained, specifically including:

[0081] Sample random vectors from a preset standard normal distribution space;

[0082] Through the feature fusion module, the random vector and the speech feature are fused to obtain a feature fusion vector;

[0083] The feature fusion vector is input into the decoding module for decoding to obtain multiple reference key point data.

[0084] Specifically, see Figure 4 , Figure 4 : is a schematic diagram of the facial key point prediction module 220 provided in an embodiment of the present application. The facial key point prediction module 220 includes a feature fusion module 221 and a decoding module 222 during actual use. Among them, the feature fusion module 221 is used to combine the Gaussian prior distribution results of the facial key points obtained by the facial key point prediction module 220 during the training process to predict the facial key points corresponding to the target voice; the decoding module 222 is used to decode the fused feature vector to obtain a plurality of predicted reference key points.

[0085] In the examples of this application, see Figure 5 , Figure 5 : It is a schematic diagram of the training process of the facial key point prediction module 220 provided in an embodiment of the present application. During the training process, the facial key point prediction module 220 can be composed of an encoding module 223, a feature fusion module 221 and a decoding module 222, with the input being training video data and the output being predicted facial key points. The facial key point prediction module 220 is essentially a variational autoencoder (VAE), and the goal of the variational autoencoder is to learn a probability distribution rather than a single vector representation. Among them, the training data used in the training process are the Chinese and English videos mentioned in the aforementioned steps. The training data can be obtained from a training video database or downloaded from the Internet, and the acquisition method is not limited.

[0086] Furthermore, 3DMM face key point detection is performed on the face in the training video data to obtain the face key points in the training video. In this embodiment, the five chin key points and 20 mouth key points in the 3DMM face key points can be used for self-encoding training to avoid the influence of face posture information, and the mouth key points are trained to improve the correlation between the voice features and the mouth key points.

[0087] Furthermore, the detected 3DMM face key points are input to the encoding module 223 for encoding. The encoding module 223 can map the input data (e.g., the key point coordinates of the face image in the Chinese and English videos) to the latent vector in the latent space. The vector in the latent space is low-dimensional and can capture the main features of the input data. Then, the encoding module 223 can output two vectors, which are used to represent the distribution of the latent vector. The probability distribution of the latent vector is represented by the mean μ and variance σ of the latent vector, that is, the Gaussian prior distribution result of the latent vector can be obtained.

[0088] After obtaining the Gaussian prior distribution results of facial key points in the above steps, the speech features extracted from 200 hours of Chinese and English videos are fused with the Gaussian prior distribution results output by the encoding module 223 through the feature fusion module 221 to obtain a feature fusion vector. Further, the feature fusion vector is input to the decoding module 222, and the decoding module 222 reconstructs the feature fusion vector back to the original data space to obtain the facial key points predicted based on the speech features. In this way, by training the facial key point prediction module 220, the facial key point prediction module 220 can learn the ability to predict facial key points based on speech features, and learn the ability to generate new facial key points other than the faces in the training video data.

[0089] Based on the above steps, the facial key point prediction module 220 is trained. Further, the facial key point prediction module 220 can predict the facial key points used to drive the target object based on the speech features of the target speech. The specific prediction process is: a vector can be randomly sampled from the normal distribution Z~N(0,1), and the sampled random vector and the speech features of the target speech are input into the feature fusion module 221 for feature fusion to obtain a feature fusion vector, which is a facial key point representation vector predicted by the speech features of the target speech. Further, the feature fusion vector is input into the decoding module 222 for decoding to obtain multiple reference key point data of the target object. Among them, each reference key point data corresponds to the speech features corresponding to the speech segment in a time zone of the target speech, and the reference key point data can be the key point data of the lower half of the face area of ​​the target object, specifically the chin reference key point data and the mouth reference key point data. The facial key point prediction module 220 provided in the embodiment of the present application removes the influence of facial posture in the prediction process, thereby improving the accuracy of facial key point prediction.

[0090] After obtaining the reference key point data corresponding to the speech features of the target speech, the predicted reference point data can be adjusted by the facial key points of the target object to generate the facial key points of the target object driven by the target speech. Prior to this, the facial key point data of the target object can be obtained first.

[0091] In step 104 of some embodiments, obtaining first facial key point data and second facial key point data of the target object includes:

[0092] Obtain object video of the target object;

[0093] Acquiring a reference video frame from the object video;

[0094] Perform image extraction based on the reference video frame to obtain the facial image of the object;

[0095] Performing image segmentation on the object's facial image to obtain a first facial image and a second facial image;

[0096] Extract key points from the first facial image to obtain first facial key point data;

[0097] Key point extraction is performed on the second facial image to obtain second facial key point data.

[0098] Specifically, the first facial key point data is used to represent the key point data that is the same as the reference key point data in the facial area of ​​the target object, that is, the first facial key point represents the facial image corresponding to the mouth area of ​​the target object, and may include chin key point data and mouth key point data; the second facial key point data is used to represent the key point data that is different from the reference key point data in the facial area of ​​the target object, that is, the second facial key point represents the facial image outside the mouth area of ​​the target object, and may include the key point data of the upper half of the facial area of ​​the target object, such as eyebrow key point data and nose key point data.

[0099] In this embodiment, the object video of the target object can be obtained first. The object video is a real-life video provided by the target object, and the duration can be one minute to one and a half minutes. Further, the reference video frame can be collected from the object video, specifically including:

[0100] Obtain the shooting clarity and shooting completeness of each video frame in the target video;

[0101] Comparing the shooting clarity in each video frame to obtain a shooting clarity comparison result;

[0102] Comparing the shooting completeness in each video frame to obtain a shooting completeness comparison result;

[0103] Based on the shooting clarity comparison result and the shooting completeness comparison result, a reference video frame is determined in the object video.

[0104] In this embodiment, the shooting clarity and shooting completeness of each video frame in the object video can be obtained, specifically, the shooting clarity and shooting completeness of the mouth of the target user. Further, the shooting clarity in each video frame is scored, and the video frames are sorted according to the scores to obtain the shooting clarity comparison result. The smaller the pixel of the object video, the lower the resolution of each frame of the picture, and the lower the shooting clarity, the more frames of pictures need to be combined to determine the facial key point data of the virtual object; similarly, the shooting completeness in each video frame can be compared to obtain the shooting completeness comparison result.

[0105] Furthermore, the shooting clarity comparison results and the shooting completeness comparison results are comprehensively evaluated. For example, the shooting completeness comparison results can be analyzed first to obtain several frames of images with higher shooting completeness scores. Then, among the several images with higher shooting completeness scores, the images are screened according to the shooting clarity scores, and the images with the highest shooting clarity are screened out from the several images with higher shooting completeness. The disclosed embodiment obtains one or more frames of images with better shooting clarity and shooting completeness by comprehensively evaluating the shooting clarity score and the shooting completeness score of each frame of the object video, and uses the obtained one or more frames of images as reference video frames of the target object sampled from the object video. The present embodiment obtains reference video frames with higher shooting clarity and shooting completeness by screening each frame of the object video, thereby ensuring the accuracy of the facial image of the target object generated subsequently.

[0106] In other embodiments, the mouth shape key points in each frame image in the object video can also be extracted. In the aforementioned steps, when the facial key point prediction module 220 is trained, a database storing the correspondence between voice features and predicted key points can be formed according to the training video data and the corresponding voice features, and the database stores the mouth shape data corresponding to different voice features. The mouth shape key points of each frame image in the object video can be compared with the mouth shape data stored in the database, so as to ignore the repeated mouth shapes in the object video, extract several frames of images with more obvious mouth shape changes for training the face generation module 230, and generate an editable face image of the target object. In this way, repeated mouth shapes can be removed, and the face generation module 230 does not need to be trained according to each frame image in the object video, which saves training time and improves the efficiency of face image generation.

[0107] In some embodiments, after obtaining the reference video frame of the object video through the aforementioned steps, image extraction can be performed based on the reference video frame to obtain the object's facial image. Through image extraction, this embodiment can remove the influence of background and facial posture, and further improve the accuracy of facial image generation of the target object. Further, the object's facial image is divided into images to obtain a first facial image and a second facial image, the first facial image represents the image corresponding to the mouth area, including images of the chin and mouth areas; the second facial image represents the image of the facial area other than the mouth area. After separating the first facial image and the second facial image, 3DMM key point detection can be performed on the first facial image and the second facial image respectively to obtain first facial key point data and second facial key point data. Among them, the first facial key point data is used to represent the changes in the mouth shape key points, and the second facial key point data is used to represent the changes in the eyebrow key points.

[0108] In step 105 of some embodiments, the aforementioned steps predict a plurality of reference key point data corresponding to the target speech according to the speech feature of the target speech. Among them, the plurality of reference key point data are key point data corresponding to the speech feature of the target speech and applicable to the faces of all target objects, i.e., a universal face key point data template, which can be used for different target objects when the same speech is used for driving. However, since different target objects have different facial features, such as face shape, chin shape, mouth shape change habits when speaking, and other facial detail features, the mouth shape generated using the universal template may not necessarily meet the speaking habits, speaking state, and some facial detail representations of the target object. In order to make the generated face closer to the target object, the universal key point data template can be fine-tuned so that the fine-tuned face key point data matches the facial features of the target object, and the generated face image can more accurately reflect the expression, mouth shape change, and speaking habits of the target object, so as to improve the accuracy and fidelity of the generated face.

[0109] Based on this, the first facial key point data extracted from the reference video frame of the target video can be used to adjust the multiple reference point data to obtain the target facial key point data, so that the predicted facial key point data can adapt to the facial feature changes of the target object when speaking, and adjust for the personalized differences in mouth shape changes between different target objects. Specifically, the data adjustment of the multiple reference key point data based on the first facial key point data to obtain the target facial key point data includes but is not limited to the following steps:

[0110] Determine a first facial key point data mean and a first facial key point data variance according to the first facial key point data;

[0111] Determine the reference key point data mean and the reference key point data variance according to the plurality of reference key point data;

[0112] Based on the first facial key point data mean, the first facial key point data variance, the reference key point data mean and the reference key point data variance, each reference key point data is normalized to obtain the target facial key point data.

[0113] In this embodiment, the data of the multiple reference key point data is adjusted based on the first facial key point data, and the target facial key point data is obtained by normalizing the multiple reference key point data. Specifically, if the data of the multiple reference key point data is adjusted based on the first facial key point data during the training process of the facial key point prediction module 220, the specific normalization formula is as follows:

[0114]

[0115] Among them, X represents the first facial key point data, represents the mean value of the first facial key point data, σ X represents the variance of the first facial key point data, represents the mean of multiple reference key point data, σ Y represents the variance of multiple reference key point data, and Z represents the target facial key point data after normalization and transformation to the specific expression domain of the target object. Wherein, the first facial key point data and the second facial key point data are both represented by arrays, and the target facial key point data represents the facial key point data of the target object when driven by the target voice, specifically the mouth key point data.

[0116] The embodiment of the present application converts multiple reference key point data into the specific expression domain of the target object through normalization, so that the reference key point data can better express the mouth features of the target object when speaking, improve the prediction accuracy of the mouth key points of the target object, and thereby improve the accuracy of the mouth shape in the generated target face video.

[0117] In step 106 of some embodiments, generating a face video based on the target facial key point data and the second facial key point data to obtain the target face video includes but is not limited to the following steps:

[0118] The reference video frame and the target facial key point data are encoded by a preset encoder to obtain a target feature vector and a target bias vector;

[0119] By using a preset face image generator, an image is generated based on a target feature vector and a target bias vector to obtain a target mouth shape adjustment image;

[0120] generating a target facial image excluding a mouth area of ​​the target object according to the second facial key point data;

[0121] The target mouth shape adjustment image and the target facial image are synthesized into a video to generate a target face video.

[0122] Specifically, see Figure 6 , Figure 6 2 is a schematic diagram of a face generation module 230 provided in an embodiment of the present application. The face generation module 230 includes an encoder 231 and a face image generator 232. The input of the face generation module 230 is a reference video frame and target facial key point data, and the face image generator 232 is a model trained by multiple high-definition face image data, and has the ability to generate face images. Figure 6In the example, an image with six small flowers as the background is used as the reference video frame. A black mask image is generated based on the target facial key point data. The reference video frame is masked to remove most of the background information and posture information of the portrait. This allows the model to focus only on the mouth area when generating the target object's face image, thereby accelerating the convergence speed and shortening the time to generate the target face video. The model output is a mouth image with the mouth shape adjusted and the background image removed. Figure 6 It can be seen that the output image does not contain small flowers, indicating that the background has been removed.

[0123] In this embodiment, the masked reference video frame is encoded by the encoder 231 to obtain a target feature vector W and a target bias vector Δ, wherein there are eighteen target bias vectors, namely 18*Δ, and the target bias vector is used to adjust the details of the generated face image. The masked reference video frame is the mouth mask image of the target object. The encoder 231 extracts the key features of the mask image and converts these features into a vector representation W in the latent space and eighteen target bias vectors Δ.

[0124] Furthermore, the target feature vector and the target offset are added, and the vector result obtained by the addition is input into the face image generator 232 to generate a face image, thereby obtaining a target mouth shape adjustment image. Figure 6 It can be seen that the output of the face generation module 230 is an image after the mouth shape is adjusted for the mouth area of ​​the target object.

[0125] In some embodiments, the face image generator 232 is pre-trained, and the face image generator 232 can generate an editable face image of the target object to facilitate the adjustment of its mouth shape. Among them, the face image generator 232 can generate a universal face image template. In actual use, the face generation module 230 can be trained according to the mask image of the reference video frame to fine-tune the parameters of the face image generator 232, so that a face image that is closer to the target object can be generated; the parameters of the encoder 231 can also be adjusted according to the mask image of the reference video frame, so that the generated face image can be controlled by the target facial key points to generate the mouth shape corresponding to the target voice.

[0126] The training process of the face generation module 230 is divided into two stages, see Figure 7 , Figure 7 2 is a schematic diagram of the first training stage of the face generation module 230 provided in the embodiment of the present application. Specifically, the first training stage is the pivot fine-tuning inversion stage, the goal of which is to generate the mouth area of ​​the target face image, that is, to fine-tune (unfreeze) the parameters of the face image generator 232 and keep the parameters of the encoder 231 unchanged (freeze), so that the generated face image is closer to the user's real face.

[0127] Specifically, the target object face image is encoded by the encoder 231 to generate a W vector and 18 bias vectors, which are added and input into the face image generator 232 for face image generation. The generated face image is an image of the mouth area with the background removed. Among them, the mouth shape in the input target face image is an "O" shape, with small flowers in the background; the output is the generated mouth area image with the background removed, and the mouth shape is also an "O" shape. Furthermore, the loss function of the first training stage can be calculated, and the parameters of the face image generator 232 can be fine-tuned according to the loss function. The specific expression of the loss function of the first training stage is as follows:

[0128] L 阶段一 =L LPIPS (x,x p )+λL L2 (x,x p )

[0129] Among them, the learned perceptual image patch similarity loss (Learned Perceptual Image PatchSimilarity, LPIPS) loss, namely L LPIPS , is a loss function used to measure the perceptual similarity between two images, reducing L LPIPS The purpose is to enable the face image generator 232 to generate a face image that is more similar to the real face of the target object in visual perception. In addition, the loss function of the first training stage also includes L2 loss, which is essentially a mean squared error (MSE) loss, which is used to measure the pixel-level difference between the generated face image and the target face image. L LPIPS With L L2 The input is the generated mouth image x p and the mouth image x of the target object, λ is the weight parameter.

[0130] In some embodiments, the second training phase of the face generation module 230 is a mouth shape matching phase, see Figure 8 , Figure 8 2 is a schematic diagram of the second training stage of the face generation module provided in the embodiment of the present application. In the second training stage, the input of the face generation module 230 is the face image of the target object after the mouth is masked, and the output is the mouth area image after the mouth shape is adjusted in the first training stage. Figure 8It can be seen that the mouth shape of the target object changes from "O" shape to "open smile" mouth shape. The purpose of the second training stage is to adjust the parameters of the encoder 231 obtained by the first stage training, and the parameters of the face image generator 232 remain unchanged, so that the generated mouth area image can be controlled by the target facial key points. The loss function formula corresponding to the second training stage is as follows:

[0131] L 阶段二 =L LPIPS (x,x p )+αL L2 (x,x p )+βL ID (x,x p )

[0132] Among them, L LPIPS and L L2 This has been introduced in the previous steps and will not be repeated here. x is the input target object's mouth mask image, x p To adjust the image of the generated target mouth shape, it is the mouth area mask loss function L ID , used for mask loss calculation to generate the difference between the mouth area of ​​the image and the mouth area of ​​the target image, α and β are set weight coefficients, which can be flexibly set according to actual needs. By adjusting the parameters of the encoder 231, the training loss in the second training stage is continuously reduced.

[0133] In the training process of the disclosed embodiment, the mouth area in the face image of the target object can be masked, so that the influence of the background can be removed. The face image generator 232 is trained only through the mouth mask area, which can effectively speed up the convergence speed of the face image generator 232. Compared with the related art method of face generation based on the entire facial area, the present application can shorten the training time to 15 to 60 minutes. In addition, only fine-tuning the key points of the mouth can improve the generation efficiency of the target adjusted mouth shape image, thereby improving the generation efficiency of the target face video.

[0134] In the above steps, the face generation module 230 is trained by a two-stage training method. The face generation module 230 is trained based on the face image of the target object, and generates a mouth area image that is closest to the target object by continuously fine-tuning the model parameters.

[0135] Furthermore, after the target mouth shape adjustment image is generated in the above step, a target facial image other than the mouth area of ​​the target object may be generated according to the second facial key point data. Generating the target facial image other than the mouth area of ​​the target object according to the second facial key point data specifically includes:

[0136] Extract emotional features from the target speech to obtain emotional features;

[0137] Classify the emotional features based on the preset emotional classification model to obtain the emotional state of the target object;

[0138] Adjusting the key point positions of the second facial key point data according to the emotional state of the target object to generate facial key point data corresponding to the emotional state of the target object;

[0139] An image is generated based on the facial key point data corresponding to the emotional state of the target object to obtain an image of the target facial area excluding the mouth area of ​​the object.

[0140] In this embodiment, emotional features can be extracted from the target speech, wherein the emotional features represent speech features in the target speech that can reflect emotional changes, such as changes in speech speed, changes in pitch, changes in sound intensity, etc. Further, the emotional features are classified based on a preset emotion classification model to determine the emotional state of the object. The emotion classification model can be any type of deep learning model that is trained with speech data with emotion labels. The output of the emotion classification model is an emotional state probability distribution, which indicates the probability that the input emotional features belong to different emotion categories. The emotional state of the object is determined based on the emotion category corresponding to the highest probability.

[0141] Furthermore, the key point positions of the first facial key point data and the second facial key point data may be adjusted according to the emotional state of the object, and the facial key point data corresponding to the emotional state of the object may be generated to simulate the expression of the target object in the current emotional state. For example, if the emotional state of the object is classified as "happy", the key point positions of the corners of the mouth may be moved upward to simulate a smiling expression, and the key point positions of the eyes may be adjusted accordingly to present a curved state of the eyes when smiling.

[0142] After generating facial key point data corresponding to the emotional state of the object, an image can be generated based on the facial key point data to obtain a target facial area image other than the mouth area of ​​the object. The target facial area image is spliced ​​with the target mouth shape adjustment image to obtain a target object face image. According to the time zone sequence of the target voice data, the generated target object face image is fused with each frame of the background image corresponding to each time zone, and the video frames corresponding to all time zones are fused to obtain the target face video.

[0143] See also Fig. 9 The embodiment of the present application further provides a face video generation device 900, which can implement the above-mentioned face video generation method, and the device includes:

[0144] A first acquisition unit 910 is used to acquire a target voice for driving a target object to utter, wherein the target voice includes a plurality of continuous time zones, each of which corresponds to a frame in a target face video of the target object;

[0145] An extraction unit 920, configured to extract features from the target speech to obtain speech features;

[0146] A first generating unit 930 is used to input the speech feature into a pre-trained facial key point prediction module to generate key points, and obtain a plurality of reference key point data, each of which corresponds to the speech feature of a time zone of the target speech;

[0147] A second acquisition unit 940 is used to acquire first facial key point data and second facial key point data of the target object, wherein the first facial key point data is used to represent key point data that is the same as the reference key point data in the facial region of the target object, and the second facial key point data is used to represent key point data that is different from the reference key point data in the facial region of the target object;

[0148] An adjusting unit 950 is used to adjust the plurality of reference key point data based on the first facial key point data to obtain target facial key point data;

[0149] The second generating unit 960 is used to generate a face video based on the target facial key point data and the second facial key point data to obtain the target face video.

[0150] The specific implementation of the face video generation device is basically the same as the specific implementation of the above-mentioned face video generation method, and will not be repeated here.

[0151] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned face video generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0152] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0153] The processor 1001 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0154] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the face video generation method of the embodiment of this application;

[0155] Input / output interface 1003, used to implement information input and output;

[0156] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0157] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0158] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0159] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned face video generation method is implemented.

[0160] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0162] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0163] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0165] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0166] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0171] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for generating a face video, characterized in that: The method comprises: Acquire a target voice for driving a target object to utter, the target voice comprising a plurality of continuous time zones, each of the time zones corresponding to a frame in a target face video of the target object; Extracting features of the target speech to obtain speech features; Inputting the speech feature into a pre-trained facial key point prediction module to generate key points, and obtaining a plurality of reference key point data, each of which corresponds to the speech feature of a time zone of the target speech; Acquire first facial key point data and second facial key point data of the target object, wherein the first facial key point data is used to represent key point data that is the same as the reference key point data in the facial region of the target object, and the second facial key point data is used to represent key point data that is different from the reference key point data in the facial region of the target object; Performing data adjustment on the plurality of reference key point data based on the first facial key point data to obtain target facial key point data; A facial video is generated based on the target facial key point data and the second facial key point data to obtain the target facial video.

2. The method according to claim 1, characterized in that The step of acquiring the first facial key point data and the second facial key point data of the target object comprises: Acquire an object video of the target object; Acquiring a reference video frame from the object video; Perform image extraction based on the reference video frame to obtain a facial image of the object; Performing image segmentation on the facial image of the target object to obtain a first facial image and a second facial image, wherein the first facial image represents a facial image corresponding to a mouth area of ​​the target object, and the second facial image represents a facial image outside the mouth area of ​​the target object; Extracting key points from the first facial image to obtain first facial key point data; Key point extraction is performed on the second facial image to obtain second facial key point data.

3. The method according to claim 2, characterized in that The acquiring of reference video frames from the object video comprises: Obtaining the shooting clarity and shooting completeness of each video frame in the object video; Comparing the shooting clarity in each of the video frames to obtain a shooting clarity comparison result; Comparing the shooting completeness in each of the video frames to obtain a shooting completeness comparison result; Based on the shooting clarity comparison result and the shooting completeness comparison result, a reference video frame is determined in the object video.

4. The method according to claim 1, characterized in that: The facial key point prediction module includes a feature fusion module and a decoding module. The speech feature is input into the pre-trained facial key point prediction module to generate key points, and a plurality of reference key point data are obtained, including: Sample random vectors from a preset standard normal distribution space; By means of the feature fusion module, the random vector and the speech feature are subjected to feature fusion to obtain a feature fusion vector; The feature fusion vector is input into the decoding module for decoding to obtain a plurality of reference key point data.

5. The method according to claim 1, characterized in that The step of adjusting the plurality of reference key point data based on the first facial key point data to obtain target facial key point data includes: Determine a first facial key point data mean and a first facial key point data variance according to the first facial key point data; Determine a mean value of the reference key point data and a variance of the reference key point data according to the plurality of reference key point data; Based on the first facial key point data mean, the first facial key point data variance, the reference key point data mean and the reference key point data variance, each of the reference key point data is normalized to obtain target facial key point data.

6. The method according to claim 2, characterized in that The generating of a face video based on the target face key point data and the second face key point data to obtain the target face video includes: Encoding the reference video frame and the target facial key point data by a preset encoder to obtain a target feature vector and a target offset vector; By using a preset face image generator, an image is generated based on the target feature vector and the target bias vector to obtain a target mouth shape adjustment image; Generate a target facial image excluding a mouth area of ​​the target object according to the second facial key point data; The target mouth shape adjustment image and the target facial image are subjected to video synthesis to generate the target face video.

7. The method according to claim 6, characterized in that The step of generating a target facial image outside the mouth area of ​​the target object according to the second facial key point data comprises: Extracting emotional features from the target speech to obtain emotional features; Classifying the emotional features based on a preset emotional classification model to obtain the emotional state of the target object; Adjusting the key point positions of the second facial key point data according to the emotional state of the target object to generate facial key point data corresponding to the emotional state of the target object; An image is generated based on the facial key point data corresponding to the emotional state of the target object to obtain an image of the target facial area excluding the mouth area of ​​the object.

8. A face video generation device, characterized in that: The face video generating device comprises: A first acquisition unit is used to acquire a target voice for driving a target object to emit, wherein the target voice includes a plurality of continuous time zones, each of which corresponds to a frame in a target face video of the target object; An extraction unit, used for extracting features from the target speech to obtain speech features; A first generating unit is used to input the speech feature into a pre-trained facial key point prediction module to generate key points, and obtain a plurality of reference key point data, each of which corresponds to the speech feature of a time zone of the target speech; a second acquisition unit, configured to acquire first facial key point data and second facial key point data of the target object, wherein the first facial key point data is used to represent key point data that is the same as the reference key point data in the facial region of the target object, and the second facial key point data is used to represent key point data that is different from the reference key point data in the facial region of the target object; an adjusting unit, configured to adjust the plurality of reference key point data based on the first facial key point data to obtain target facial key point data; The second generating unit is used to generate a face video based on the target facial key point data and the second facial key point data to obtain the target face video.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the face video generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the face video generation method according to any one of claims 1 to 7 is implemented.

Images