Face video reasoning method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a face video reasoning method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a target voice for driving a target object and a pre-coded face attribute latent vector; performing feature coding on the target voice through a preset voice coding module to obtain a target mouth shape latent vector; performing feature fusion on the face attribute latent vector and the target mouth shape latent vector through a pre-trained cross-domain fusion module to obtain a target latent vector; inputting the target latent vector into a pre-trained first face image generation module for image generation to obtain a target face image of the target object; and performing video reasoning according to the time zone sequence of the target voice and the multiple frames of target face images corresponding to each time zone to obtain a target face video. According to the embodiment of the invention, the reasoning efficiency of the face video can be improved, and the generated face video has relatively high face image quality.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a face video reasoning method and device, an electronic device and a storage medium. Background Art

[0002] 2D digital human refers to a digital human modeled directly on a two-dimensional plane. Usually, the user provides a real-life video or a single image, and generates a realistic spoken-word video through text or voice drive, which can reduce the labor and time costs of video creation.

[0003] 2D digital humans can customize two-dimensional virtual characters for users. Such virtual characters can be widely used in entertainment live broadcasts, news reports, corporate education and other occasions. Taking entertainment live broadcasts as an example, the virtual characters displayed during the live broadcast are required to be as close to real people as possible. In addition, the live broadcast process has very high requirements for the video reasoning speed. The video frame rate is an important indicator for evaluating the video reasoning speed, which indicates the number of frames inferred per second. Live broadcast videos with high video frame rates can provide better live viewing effects. In related technologies, the video reasoning speed is slow, resulting in a low video frame rate and poor viewing effect. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a face video reasoning method and device, an electronic device and a storage medium, aiming to improve the reasoning efficiency of face videos.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a face video inference method, the method comprising:

[0006] Acquire a target speech for driving a target object and a pre-encoded facial attribute latent vector, wherein the target speech includes a plurality of continuous time zones;

[0007] Performing feature encoding on the target speech through a preset speech encoding module to obtain a target mouth shape latent vector;

[0008] Performing feature fusion on the facial attribute latent vector and the target mouth shape latent vector through a pre-trained cross-domain fusion module to obtain a target latent vector;

[0009] Inputting the target latent vector into a pre-trained first face image generation module to generate an image, thereby obtaining a target face image of the target object;

[0010] Video inference is performed based on the time zone sequence of the target speech and multiple frames of the target face images corresponding to each time zone to obtain the target face video.

[0011] In some embodiments, the pre-coded facial attribute latent vector is generated according to a preset student model, the student model includes a first facial attribute encoding module, and the method for obtaining the pre-coded facial attribute latent vector specifically includes:

[0012] Acquire a reference video and a first mouth key point of the target object, where the first mouth key point represents a mouth key point of the target object when the mouth is closed;

[0013] Extracting multiple frames of reference images from the reference video, and masking the mouth area of ​​the target object in each of the reference images to obtain a mouth area masked image;

[0014] The mouth area mask image and the first mouth key point are input into the first face attribute encoding module for face attribute encoding to obtain a pre-encoded face attribute latent vector.

[0015] In some embodiments, the student model further includes the first face image generation module, the teacher model includes the second face image generation module and the second face attribute encoding module, the teacher model is used to guide the training of the student model, and before inputting the target latent vector into the pre-trained first face image generation module for image generation, the method further includes: training the first face image generation module, specifically including:

[0016] Extracting mouth key points from each of the reference images to obtain second mouth key points, where the second mouth key points represent the mouth key points of the target object when the mouth is opened;

[0017] Inputting the mouth area mask image and the second mouth key points into the second face attribute encoding module for face attribute encoding to obtain a guiding latent vector, wherein the guiding latent vector is used to guide the generation of a target latent vector;

[0018] Extracting a mouth area image of the target object from each of the reference images to obtain a mouth area image;

[0019] The guiding latent vector is input into the second face image generation module for image generation, the first loss is calculated with the mouth area image as the output label of the second face image generation module, and the parameters of the second face image generation module are updated according to the first loss to obtain the first face image generation module.

[0020] In some embodiments, extracting multiple reference image frames from the reference video includes:

[0021] Detecting changes in the mouth shape of the target object in the reference video to obtain mouth shape information of the target object in each frame of image;

[0022] Determine a plurality of frames of the reference image in the reference video according to the mouth shape information.

[0023] In some embodiments, before the feature fusion of the facial attribute latent vector and the target mouth shape latent vector is performed by the pre-trained cross-domain fusion module, the method further includes: training the cross-domain fusion module, specifically including:

[0024] Taking the guiding latent vector as the output label of the cross-domain fusion module, performing loss calculation according to the guiding latent vector and the target latent vector to obtain a second loss;

[0025] Parameters of the cross-domain fusion module are updated according to the second loss.

[0026] In some embodiments, the speech encoding module includes a feature extraction submodule and a cross-domain conversion submodule, and the feature encoding of the target speech by the preset speech encoding module to obtain the target mouth shape latent vector includes:

[0027] Extracting features of the target speech through the feature extraction submodule to obtain target speech features;

[0028] The target speech feature is converted into the facial attribute latent vector space of the target object through the cross-domain conversion submodule to obtain the target mouth shape latent vector.

[0029] In some embodiments, the target facial image includes a target mouth image and a facial expression image of the target object, and the step of inputting the target latent vector into a pre-trained first facial image generation module for image generation to obtain the target facial image of the target object includes:

[0030] Inputting the target latent vector into a pre-trained first face image generation module to generate an image, thereby obtaining the target mouth image;

[0031] Performing sentiment analysis on the target speech through a preset sentiment analysis module to obtain sentiment analysis features;

[0032] Cross-domain fusion of the sentiment analysis feature and the pre-encoded facial attribute latent vector to generate a facial expression latent vector of the target object;

[0033] Inputting the facial expression latent vector into the first face image generation module to generate an image, thereby obtaining a facial expression image of the target object;

[0034] The facial expression image and the target mouth image are spliced ​​to obtain the target face image.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a face video inference device, the device comprising:

[0036] An acquisition unit, used to acquire a target voice and a pre-encoded facial attribute latent vector for driving a target object, wherein the target voice includes a plurality of continuous time zones;

[0037] An encoding unit, used for performing feature encoding on the target speech through a preset speech encoding module to obtain a target mouth shape latent vector;

[0038] A fusion unit, configured to perform feature fusion on the face attribute latent vector and the target mouth shape latent vector through a pre-trained cross-domain fusion module to obtain a target latent vector;

[0039] A generating unit, configured to input the target latent vector into a pre-trained first face image generating module for image generation to obtain a target face image of the target object;

[0040] The inference unit is used to perform video inference according to the time zone sequence of the target speech and the multiple frames of the target face images corresponding to each time zone to obtain the target face video.

[0041] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0043] The face video reasoning method and device, electronic device and storage medium proposed in the present application obtain the target speech and pre-coded face attribute latent vector for driving the target object; feature encode the target speech through a preset speech coding module to obtain the target mouth shape latent vector; feature fuse the face attribute latent vector and the target mouth shape latent vector through a pre-trained cross-domain fusion module to obtain the target latent vector; input the target latent vector into the pre-trained first face image generation module for image generation to obtain the target face image of the target object; perform video reasoning based on the time zone sequence of the target speech and the multiple frames of target face images corresponding to each time zone to obtain the target face video. Since the time-consuming pre-coded face attribute latent vector has been generated in advance, in the actual face video reasoning process, it is only necessary to directly obtain the pre-coded face attribute latent vector to generate the target face image, which greatly improves the reasoning speed of the target face video. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the face video inference method provided in an embodiment of the present application;

[0045] Figure 2 is a flowchart of obtaining a pre-coded facial attribute latent vector provided by an embodiment of the present application;

[0046] Figure 3 Schematic diagram of a training module of a face video inference model provided in an embodiment of the present application;

[0047] Figure 4 is a schematic diagram of a speech coding module provided in an embodiment of the present application;

[0048] Figure 5 is a schematic diagram of a cross-domain fusion module provided in an embodiment of the present application;

[0049] Figure 6 It is a schematic diagram of the process of training the first face image generation module provided in an embodiment of the present application;

[0050] Figure 7 It is a schematic diagram of an application module of a face video inference model provided in an embodiment of the present application;

[0051] Figure 8 It is a structural schematic diagram of a face video inference device provided in an embodiment of the present application;

[0052] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] Based on this, the embodiments of the present application provide a face video reasoning method and device, an electronic device and a storage medium, aiming to improve the reasoning efficiency of face videos.

[0057] The face video inference method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the face video inference method in the embodiments of the present application is described.

[0058] The face video reasoning method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The face video reasoning method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the face video reasoning method, etc., but is not limited to the above forms.

[0059] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0060] Figure 1 is an optional flowchart of the face video inference method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps 101 to 105:

[0061] Step 101, obtaining a target voice and a pre-encoded facial attribute latent vector for driving a target object;

[0062] Step 102, feature encoding the target speech through a preset speech encoding module to obtain a target mouth shape latent vector;

[0063] Step 103, performing feature fusion on the facial attribute latent vector and the target mouth shape latent vector through a pre-trained cross-domain fusion module to obtain a target latent vector;

[0064] Step 104, inputting the target latent vector into a pre-trained first face image generation module to generate an image, thereby obtaining a target face image of the target object;

[0065] Step 105, performing video inference according to the time zone sequence of the target speech and the multiple frames of target face images corresponding to each time zone to obtain the target face video.

[0066] In step 101 of some embodiments, the embodiments of the present application can customize an exclusive virtual character image for a specific user and generate a target face video containing the virtual character image. The target object can be the virtual character image in the target face video finally generated, or it can refer to a specific user. The target voice is a voice sequence used to drive the target object. The target voice contains multiple continuous time zones, and each time zone can correspond to multiple frames of images in the target face video. The pre-encoded facial attribute latent vector is the latent vector corresponding to the facial image of each target object when the mouth is closed. The pre-encoded facial attribute latent vector can be driven by the target voice. The mouth shape of the target character in the face image is completely driven by the target voice. After the mouth shape corresponding to the target voice is generated, it can be combined with facial images of other parts to finally generate a complete facial image.

[0067] The encoding time of the face attribute encoder in the related art is relatively long, and the generation process of the face attribute latent vector takes a long time, resulting in a slow inference speed of the face video. In this embodiment, the face attribute latent vector is generated in advance during the model training process. When the target face video is actually generated, the pre-encoded face attribute latent vector can be directly used, and the face attribute latent vector is lip-encoded according to the target voice to generate the corresponding target face image. The whole process omits the tedious process of generating the face attribute latent vector, and improves the inference speed of the face video.

[0068] In some embodiments, the pre-coded facial attribute latent vector is generated during the model training process. Next, the generation process of the pre-coded facial attribute latent vector is introduced. The method for obtaining the pre-coded facial attribute latent vector specifically includes but is not limited to the following steps 201 to 203:

[0069] Step 201, obtaining a reference video and a first mouth key point of a target object;

[0070] Step 202, extracting multiple frames of reference images from the reference video, and masking the mouth area of ​​the target object in each reference image to obtain a mouth area mask image;

[0071] Step 203: Input the mouth area mask image and the first mouth key point into a first face attribute encoding module for face attribute encoding to obtain a pre-encoded face attribute latent vector.

[0072] Before specifically introducing this embodiment, the overall model training method provided by the embodiment of the present application is first summarized. The model for generating the target face video provided by the embodiment of the present application is trained in a distillation-like manner. Specifically, during the training process, a teacher model and a student model may be included, wherein the teacher model may be a non-real-time, easily trained, 2D digital human generation model capable of generating high-definition face images, and the student model may be a real-time, difficult to train, but capable of being guided by the teacher model. The model structure of the student model is basically the same as that of the teacher model, and the training results of the teacher model may be transferred to the student model through distillation technology, and the pre-coded facial attribute latent vector is generated based on the student model.

[0073] In some embodiments, please refer to Figure 3 , Figure 3 It is a schematic diagram of the training module of the face video reasoning model provided by the embodiment of the present application. It should be noted that the face video reasoning model may include all neural network models, modules, etc. used in the training stage of the embodiment of the present application, and may also include all neural network models and modules, etc. used in the practical application stage of the embodiment of the present application. In the training stage, the embodiment of the present application may include a teacher model 301 and a student model 302. The student model 302 may include a speech coding module, a cross-domain fusion module, a first face attribute coding module, and a first face image generation module. The teacher model 301 may include a second face attribute coding module and a second face image generation module. During the training process, the results of the teacher model 301 learning can be transferred to the student model 302, and the teacher model 301 can guide the student model 302. Specifically, the first face attribute coding module is obtained by training the second face attribute coding module, the first face image generation module is obtained by training the second face image generation module, and the guidance latent vector of the teacher model 301 is used to guide the target latent vector generated by the student model 302, so that the target latent vector generated by the student model 302 is closer to the guidance latent vector generated by the teacher model 301.

[0074] In step 201 of some embodiments, specifically, the reference video of the target object is a short real-life video of a user speaking provided by a specific user, and the reference video may include a background image, a complete face image, and a voice equal to the length of the video. It should be noted that the voice content in the reference video may be different from the target voice, and the voice in the reference video is used to train the model so that the model learns the ability to control the mouth shape of the target face according to the target voice. The first mouth key point may be the only mouth key point corresponding to the target object when the mouth is closed, which can be obtained from the reference video provided by the user, or extracted based on the closed mouth image. In the embodiment of the present application, the closed mouth key point is used as the input of the first face attribute encoding module, so that the pre-coded face attribute latent vector generated by the first face attribute encoding module does not provide any mouth shape information, and the mouth shape information is completely provided by the voice encoding module, so that the mouth shape and face posture and other attributes of the target object are completely decoupled, and the generation of the mouth shape is not affected by the face posture, thereby improving the accuracy of the mouth shape generation.

[0075] In some embodiments, multiple reference image frames may be extracted from a reference video, including but not limited to the following steps:

[0076] Detect the mouth shape changes of the target object in the reference video to obtain the mouth shape information of the target object in each frame image;

[0077] A plurality of reference image frames are determined in the reference video according to the mouth shape information.

[0078] Specifically, multiple reference frames of images in which the target object's mouth shape changes significantly and continuously in the reference video can be extracted, such as multiple reference frames in the process of changing from closed mouth to open mouth, or multiple reference frames corresponding to some key speaking mouth shapes that can represent the target object's key points when speaking. The key speaking mouth shapes can represent the tone, habits and emotions of the target object when speaking, such as the mouth shape when smiling, the mouth shape when laughing, or the mouth shape of making an "oh" sound, etc. By detecting the changes in the mouth shape of the target object in the reference video, the mouth shape information of the target object in each frame of the image can be obtained.

[0079] In some embodiments, the types of mouth shapes can be divided in advance, and the corresponding mouth shape type labels can be stored. In the actual extraction process, the mouth shape information detected in the above steps is compared with the labels to extract multiple reference images from the reference video. It should be noted that for the same mouth shape, multiple reference images can be selected, and the number of frames can be customized to avoid selecting too many images of the same mouth shape, resulting in a large amount of calculation and a cumbersome training process. The extraction of multiple reference images can be performed according to the time zone order of the reference video.

[0080] In step 202 of some embodiments, after extracting multiple frames of reference images, the mouth area of ​​the target object in each reference image may be masked to obtain a mouth area masked image. The mouth area masked image is a reference image with the mouth area of ​​the target object removed. The mouth area masked image allows the teacher model 301 and the student model 302 to focus on generating an image of the mouth area, ignoring interference from attributes such as facial posture, thereby improving the accuracy of the generated mouth shape.

[0081] In step 203 of some embodiments, the mouth mask area image and the first mouth key point may be input into the first face attribute encoder of the student model 302 for face attribute encoding to obtain a pre-encoded face attribute latent vector. It is understandable that the pre-encoded face attribute latent vector may generate a face image of the corresponding target object with the mouth closed, and then the mouth shape of the target object in the generated closed mouth face image may be driven by the target voice to generate a mouth shape corresponding to the target voice; in addition, the pre-encoded face attribute latent vector may also include face posture and image background information.

[0082] Based on this, during the training stage of the student model 302, the input of the speech encoding module of the student model 302 is the speech of the target object in the reference video, and the output is the target mouth shape latent vector. The input of the first facial attribute encoding module of the student model 302 is the mouth area mask image and the first mouth key point corresponding to multiple frames of reference images, and the output is the pre-encoded facial attribute latent vector.

[0083] In step 102 of some embodiments, the target speech may be feature encoded by a preset speech encoding module to obtain a target mouth shape latent vector, which specifically includes but is not limited to the following steps:

[0084] Extracting features of the target speech through the feature extraction submodule to obtain target speech features;

[0085] The target speech features are converted into the facial attribute latent vector space of the target object through the cross-domain conversion submodule to obtain the target mouth shape latent vector.

[0086] In some embodiments, see Figure 4 , Figure 4It is a schematic diagram of the speech coding module provided in an embodiment of the present application. Specifically, the speech coding module 400 may include a feature extraction submodule, a conversion extraction submodule and a cross-domain conversion submodule. Among them, the feature extraction submodule can be obtained by connecting a module consisting of multiple one-dimensional convolutions and Gaussian error linear units (Gaussian error LinearUnits, GeLU) in series, and specifically, it can be composed of seven layers of one-dimensional convolutions and Gaussian error linear units in series. The conversion extraction submodule is essentially a transformer extractor, and the conversion extraction submodule may include multiple conversion blocks, which are essentially transformer blocks. The multiple transformer blocks can be twelve transformer blocks, which are not limited here, and each transformer block may include an attention layer, a discard layer and a normalization layer. The feature extraction submodule and the conversion extraction submodule are used together to extract the target speech features of the target speech or the speech features of the speech in the reference video.

[0087] Furthermore, the cross-domain conversion submodule is essentially a cross-domain converter. The cross-domain conversion submodule may include multiple root mean square normalization layers and multiple fully connected blocks. The fully connected block is composed of multiple fully connected layers and activation functions alternately. The cross-domain conversion submodule is used to convert the domain where the extracted target speech features are located to the latent space of the face attributes. Specifically, the extracted target speech features can first pass through the root mean square normalization layer, and then the dimension of the target speech features can be compressed through the fully connected block to obtain a 512-dimensional vector. Further, the generated 512-dimensional vector passes through the root mean square normalization layer again, and then passes through multiple fully connected blocks respectively. For example, after eighteen fully connected blocks, each fully connected block outputs eighteen 512-dimensional vectors, and finally the eighteen generated 512-dimensional vectors are combined to obtain a 18X512-dimensional vector. The 18X512-dimensional vector passes through the root mean square normalization layer and then outputs the target mouth shape latent vector. The root mean square normalization layer provided in the embodiment of the present application can solve the problem of color overflow caused by gradient explosion that may occur in the actual training process of the face video inference model, thereby ensuring the stability of the training process and improving the clarity of the face in the generated face video.

[0088] In step 103 of some embodiments, the facial attribute latent vector obtained in the above steps and the target mouth shape latent vector can be feature fused through a pre-trained cross-domain fusion module to obtain a target latent vector. Since the pre-encoded facial attribute latent vector obtained by the first facial attribute encoding module in the student model 302 and the target mouth shape latent vector encoded by the speech encoding module 400 are both 18x512 dimensional vectors, based on this, the facial attribute latent vector and the target mouth shape latent vector can be feature fused through a cross-domain fusion module to obtain a 18x512 target latent vector. Among them, the facial attributes of the target object such as facial posture, skin color, and expression can be dominated by the facial attribute latent vector encoded by the first facial attribute encoding module, and the target mouth shape of the target object when speaking can be dominated by the target mouth shape latent vector encoded by the speech encoding module 400. Feature fusion of the two can obtain a complete facial latent vector of the target object, that is, a target latent vector.

[0089] In some embodiments, before performing feature fusion on the facial attribute latent vector and the target mouth shape latent vector through a pre-trained cross-domain fusion module, the method further includes: training the cross-domain fusion module, specifically including but not limited to the following steps:

[0090] The guiding latent vector is used as the output label of the cross-domain fusion module, and the loss is calculated according to the guiding latent vector and the target latent vector to obtain the second loss;

[0091] The parameters of the cross-domain fusion module are updated according to the second loss.

[0092] Since the second face image generation module of the student model 302 is trained based on the first face image generation module of the teacher model 301, the ability of the first face image generation module to generate a target face image is the same as the ability of the second face image generation module to generate a target face image, except that the latent vectors input by the two are different. Based on this, the embodiment of the present application supervises the target latent vector of the student model 302 through the guidance latent vector of the teacher model 301. Compared with directly using the high-definition face image output by the teacher model 301 as the supervisory signal of the face image output by the student model 302, since the dimension of the guidance latent vector is 18X512 and the size of the high-definition face image is 1024x1024, the embodiment of the present application can save storage resources, reduce memory consumption during training, and speed up training.

[0093] Specifically, the guidance latent vector of the teacher model 301 is used as the output label of the student model 302, and the second loss is calculated according to the guidance latent vector and the target latent vector. Among them, the mean square error loss (Mean Squared Error Loss, L2 Loss) function can be used, and other loss functions can also be used, which are not limited here. The second loss of the target latent vector and the guidance latent vector is calculated according to the following formula:

[0094] L 指导 =L L2 (c1,c2)

[0095] Among them, L 指导 represents the second loss, L L2 represents L2 loss, c1 represents target latent vector, and c2 represents guidance latent vector. Further, the parameters of speech encoding module 400 and cross-domain fusion module can be updated according to the value of the second loss.

[0096] During the training process of the embodiment of the present application, the input of the teacher model 301 is the mouth area mask image and the second mouth key point, and the input of the student model 302 is the mouth area mask image and the first mouth key point and the target speech which are the same as the input of the teacher model 301. Therefore, the inputs of the teacher model 301 and the student model 302 are of different modalities. The cross-domain fusion model provided by the embodiment of the present application can realize cross-modal supervision of the teacher model 301 by the student model 302.

[0097] In some embodiments, see Figure 5 , Figure 5 It is a schematic diagram of the cross-domain fusion module provided in an embodiment of the present application. After the mouth area mask image is encoded by the first facial attribute encoding module, a pre-encoded facial attribute latent vector of 18X512 dimensions is obtained. After the target speech passes through the speech encoding module 400, a target mouth shape vector of 18X512 dimensions is obtained. The target mouth shape latent vector and the facial attribute latent vector are respectively input into the fully connected block of the cross-domain fusion module 500 for dimensional compression. After passing through the fully connected block, the facial attribute latent vector is activated by the Sigmoid Gated Linear Unit (SiLU) activation function. Other activation functions can also be used here. After the facial attribute latent vector is activated, it is compressed again by the fully connected layer. The face attribute latent vector after dimensional compression is multiplied by the target latent vector after dimensional compression. The result of the vector multiplication is reduced in dimension by the fully connected block again. The reduced dimensionality vector is added to the original facial attribute latent vector to obtain the target latent vector. By Figure 5 It can be seen that the input of the cross-domain fusion module 500 is the facial attribute latent vector corresponding to the mouth area mask image and the target mouth shape latent vector corresponding to the target voice, both of which are vectors of 18X512 dimension, but they belong to features of different modalities. The cross-domain fusion module 500 provided in the embodiment of the present application can perform feature alignment and feature fusion of multi-modal features, and finally output a target latent vector that combines facial attribute features and mouth shape features. The dimension of the target latent vector is the same as that of the input latent vector, both of which are 18X512 dimensions.

[0098] In step 104 of some embodiments, the cross-domain fused target latent vector can be input into a pre-trained first face image generation module for image generation to obtain a target face image of the target object corresponding to the target voice. Prior to this, the first face image generation module can be trained, specifically including but not limited to the following steps 601 to 604:

[0099] Step 601, extracting mouth key points from each reference image to obtain second mouth key points, where the second mouth key points represent the mouth key points of the target object when the mouth is open;

[0100] Step 602, inputting the mouth area mask image and the second mouth key point into a second face attribute encoding module for face attribute encoding to obtain a guidance latent vector;

[0101] Step 603, extracting the mouth area image of the target object of each reference image to obtain the mouth area image;

[0102] Step 604: input the guidance latent vector to the second face image generation module for image generation, calculate the first loss with the mouth area image as the output label of the second face image generation module, and update the parameters of the second face image generation module according to the first loss to obtain the first face image generation module.

[0103] In step 601 of some embodiments, after extracting multiple reference images from the reference video in the above step, the mouth key points of each reference image may be extracted, for example, the key points of the three-dimensional morphable model (3D morphable model, 3DMM) of the reference image are extracted to obtain the second mouth key points. The second mouth key points are the mouth key points of the target object when speaking with the mouth open in each reference image. If there is a reference image of the target object with the mouth closed in the multiple frames, the second mouth key points may also be the mouth key points of the target object when the mouth is closed.

[0104] In step 602 of some embodiments, Figure 3 The mouth area mask image and the second mouth key point can be input into the second face attribute encoding module of the teacher model 301 for face attribute encoding to obtain a guiding latent vector. The guiding latent vector is used to guide the generation of the target latent vector, which has been introduced in the previous steps and will not be repeated here.

[0105] In step 603 of some embodiments, when performing a mouth area mask on each frame of reference image, it is necessary to extract the mouth area image in each frame of image to obtain the mouth area image corresponding to each reference image. The mouth area image can be an image of the mouth area of ​​the target object, removing the influence of facial posture and background information.

[0106] Furthermore, in step 604 of some embodiments, the guidance latent vector can be input into the second face image generation module of the teacher model 301 for image generation to obtain a predicted face image, which is a high-definition face image of size 1024X1024. The second face image generation module is a StyleGAN generator that is trained with multiple high-definition image samples and has the ability to generate high-definition face images. However, for a specific face of the target object, the parameters of the second face image generation module can also be fine-tuned to make the generated face image closer to the real face of the target object. Based on the reference video data provided by the target object, the teacher model 301 reconstructs the face image of the target object by combining the second face attribute encoding module and the second face image generation module, and reduces the reconstruction error of the face image by fine-tuning the second face image generation module.

[0107] Specifically, the second face image generator uses the mouth area image corresponding to each frame of the reference image as the output label, and calculates the first loss between the mouth area image and the predicted face image. The loss function used in the embodiment of the present application may include a learned perceptual image patch similarity (LPIPS) loss function and an L2 loss function. The LPIPS loss function is a visual perception loss function, and other forms of loss functions may also be used, which are not limited here. Specifically, the specific expression of the first loss for optimizing the second face image generation module is as follows:

[0108] L optimization = LLPIPs(x,xp) + λLL2(x,xp)

[0109] Among them, L LPIPS represents the learning-based perceptual image patch similarity loss function, L L2 represents the mean square error loss function, x represents the mouth area image, x p Represents the predicted face image, and λ represents the preset weight parameter.

[0110] Further, the parameters of the second face image generation module are updated according to the first loss, and after the parameter update is completed, the first face image generation module of the student model 302 is obtained.

[0111] In some embodiments, see Figure 7 , Figure 7It is a schematic diagram of the application module of the face video inference model provided in the embodiment of the present application. Specifically, in the actual application process of the face video inference model, the teacher model 301 is not needed. The target speech that drives the target object can be input into the speech encoding module 400 to obtain the target mouth shape latent vector, and the face attribute latent vector is obtained from the pre-encoded face attribute latent vector library according to the frame order of the reference video. The face attribute latent vector and the target mouth shape latent vector corresponding to each frame of the reference image are input into the cross-domain fusion module 500 for feature alignment and feature fusion to obtain the target latent vector. Further, the target latent vector is input into the first face image generation module to finally generate the target mouth image of the target face image. If the total duration of the multiple frames of reference images extracted from the reference video is less than the duration of the target speech, the multiple frames of reference images can be repeated to generate the target face image corresponding to each time zone in the target speech in sequence.

[0112] Since the process of generating pre-encoded face attribute latent vectors by the face attribute encoder is quite time-consuming, in actual application, it will lead to slow inference speed and low inference frame rate of the target face video. In the training process of the face video inference model, the embodiment of the present application generates pre-encoded latent vectors in advance. In actual use, only the target voice driving the target object needs to be input to perform face video inference, thereby improving the inference efficiency of face video.

[0113] In some embodiments, in addition to generating a target mouth image of the target object, the complete target face image may also include a facial expression image of the target object, and the target latent vector is input into a pre-trained first face image generation module for image generation to obtain the target face image of the target object, specifically including but not limited to the following steps:

[0114] Inputting the target latent vector into the pre-trained first face image generation module to generate an image, thereby obtaining a target mouth image;

[0115] Perform sentiment analysis on the target speech through a preset sentiment analysis module to obtain sentiment analysis features;

[0116] The emotion analysis features and the pre-encoded facial attribute latent vectors are cross-domain fused to generate the facial expression latent vector of the target object;

[0117] Inputting the facial expression latent vector into the first face image generation module to generate an image, thereby obtaining a facial expression image of the target object;

[0118] The facial expression image and the target mouth image are spliced ​​to obtain the target face image.

[0119] Specifically, the target latent vector is input into the pre-trained first face image generation module for image generation, and the method for obtaining the target mouth image has been introduced in the previous steps and will not be repeated here. Furthermore, emotional features can be extracted from the target speech through a pre-trained sentiment analysis module, which can recognize emotional information in the speech, such as happiness, sadness, anger, etc. The sentiment analysis module can be based on deep learning technology, such as Recurrent Neural Network (RNNs), Long Short-Term Memory Network (LSTMs) or other models suitable for processing time series data, which are not limited here.

[0120] Similar to the process of generating the target latent vector corresponding to the target mouth image, the emotion analysis features and the pre-coded facial attribute latent vector are cross-domain fused through the cross-domain fusion module 500 to generate the facial expression latent vector of the target object, which may include the facial expression information of the target object under different emotional states. The facial expression latent vector is input into the first face image generation module for image generation to obtain the facial expression image of the target object, and then the facial expression image and the mouth expression image of the target object can be image spliced ​​to obtain a complete target face image. Since the facial expression image and the mouth expression image correspond to the facial posture in each frame of the reference image, the original facial posture can be maintained when performing image splicing, so that the generated target face image is closer to the real image of the target object, thereby improving the quality and authenticity of the face image generation.

[0121] In step 105 of some embodiments, video inference is performed according to the time zone sequence of the target speech and the multiple frames of target face images corresponding to each time zone to obtain the target face video. After the target face image is obtained in the aforementioned step, the generated target face image can be spliced ​​with the background image in the reference image to ensure the authenticity of the picture, and the required background image can be automatically generated. The target speech can include multiple continuous time zones, each time zone corresponds to multiple frames of target face images, and the number of frames of the target face corresponding to each time zone can be defined by oneself. Further, according to the time zone sequence of the target speech, video synthesis is performed according to the target face image corresponding to each time zone, and finally the target face video is generated.

[0122] The embodiment of the present application is a solution for face video inference using pre-coded face attribute latent vectors corresponding to a single-frame reference image. By generating pre-coded face attribute latent vectors in advance, face video can be inferred efficiently, accurately, and in real time to meet video requirements in live broadcast and other fields.

[0123] According to the actual use results, the embodiment of the present application provides a method for generating high-quality digital face videos at an inference speed of 30 frames per second, which meets the requirements of live video for video quality and video frame rate. In addition, the resolution of the digital face in the generated face video is as high as 1024x1024, and the video can show delicate and realistic textures, such as hair and teeth. The mouth shape of the target object in the face video has a high matching degree and synchronization rate with the target voice. In the double-blind test, the voting rate of the generated target face video is 53%, and the voting rate of the real person video is 47%, which is very close.

[0124] See also Figure 8 The embodiment of the present application further provides a face video inference device, which can implement the above face video inference method. The face video inference device 800 includes:

[0125] An acquisition unit 801 is used to acquire a target speech and a pre-encoded facial attribute latent vector for driving a target object, wherein the target speech includes multiple continuous time zones; an encoding unit 802 is used to perform feature encoding on the target speech through a preset speech encoding module to obtain a target mouth shape latent vector; a fusion unit 803 is used to perform feature fusion on the facial attribute latent vector and the target mouth shape latent vector through a pre-trained cross-domain fusion module to obtain a target latent vector; a generation unit 804 is used to input the target latent vector into a pre-trained first face image generation module for image generation to obtain a target face image of the target object; an inference unit 805 is used to perform video inference according to the time zone sequence of the target speech and the multiple frames of target face images corresponding to each time zone to obtain a target face video.

[0126] The specific implementation of the face video inference device is basically the same as the specific implementation of the above-mentioned face video inference method, and will not be repeated here.

[0127] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned face video inference method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0128] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0129] The processor 901 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0130] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the face video inference method of the embodiment of this application;

[0131] Input / output interface 903, used to implement information input and output;

[0132] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0133] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0134] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0135] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned face video inference method is implemented.

[0136] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0138] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0139] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0141] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0142] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0144] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0146] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0147] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A face video inference method, characterized in that: The method comprises: Acquire a target speech for driving a target object and a pre-encoded facial attribute latent vector, wherein the target speech includes a plurality of continuous time zones; Performing feature encoding on the target speech through a preset speech encoding module to obtain a target mouth shape latent vector; Performing feature fusion on the facial attribute latent vector and the target mouth shape latent vector through a pre-trained cross-domain fusion module to obtain a target latent vector; Inputting the target latent vector into a pre-trained first face image generation module to generate an image, thereby obtaining a target face image of the target object; Video inference is performed based on the time zone sequence of the target speech and multiple frames of the target face images corresponding to each time zone to obtain the target face video.

2. The method according to claim 1, characterized in that The pre-coded face attribute latent vector is generated according to a preset student model, wherein the student model includes a first face attribute encoding module. The method for obtaining the pre-coded face attribute latent vector specifically includes: Acquire a reference video and a first mouth key point of the target object, where the first mouth key point represents a mouth key point of the target object when the mouth is closed; Extracting multiple frames of reference images from the reference video, and masking the mouth area of ​​the target object in each of the reference images to obtain a mouth area masked image; The mouth area mask image and the first mouth key point are input into the first face attribute encoding module for face attribute encoding to obtain a pre-encoded face attribute latent vector.

3. The method according to claim 2, characterized in that The student model further includes the first face image generation module, the teacher model includes the second face image generation module and the second face attribute encoding module, the teacher model is used to guide the training of the student model, and before the target latent vector is input into the pre-trained first face image generation module for image generation, the method further includes: training the first face image generation module, specifically including: Extracting mouth key points from each of the reference images to obtain second mouth key points, where the second mouth key points represent the mouth key points of the target object when the mouth is opened; Inputting the mouth area mask image and the second mouth key points into the second face attribute encoding module for face attribute encoding to obtain a guiding latent vector, wherein the guiding latent vector is used to guide the generation of a target latent vector; Extracting a mouth area image of the target object from each of the reference images to obtain a mouth area image; The guiding latent vector is input into the second face image generation module for image generation, the first loss is calculated with the mouth area image as the output label of the second face image generation module, and the parameters of the second face image generation module are updated according to the first loss to obtain the first face image generation module.

4. The method according to claim 2, characterized in that: The step of extracting multiple reference image frames from the reference video comprises: Detecting changes in the mouth shape of the target object in the reference video to obtain mouth shape information of the target object in each frame of image; Determine a plurality of frames of the reference image in the reference video according to the mouth shape information.

5. The method according to claim 3, characterized in that: Before the feature fusion of the face attribute latent vector and the target mouth shape latent vector is performed by the pre-trained cross-domain fusion module, the method further includes: training the cross-domain fusion module, specifically including: Taking the guiding latent vector as the output label of the cross-domain fusion module, performing loss calculation according to the guiding latent vector and the target latent vector to obtain a second loss; Parameters of the cross-domain fusion module are updated according to the second loss.

6. The method according to claim 1, characterized in that The speech encoding module includes a feature extraction submodule and a cross-domain conversion submodule. The feature encoding of the target speech is performed by the preset speech encoding module to obtain a target mouth shape latent vector, including: Extracting features of the target speech through the feature extraction submodule to obtain target speech features; The target speech feature is converted into the facial attribute latent vector space of the target object through the cross-domain conversion submodule to obtain the target mouth shape latent vector.

7. The method according to claim 1, characterized in that The target face image includes a target mouth image and a facial expression image of the target object, and the target latent vector is input into a pre-trained first face image generation module for image generation to obtain the target face image of the target object, including: Inputting the target latent vector into a pre-trained first face image generation module to generate an image, thereby obtaining the target mouth image; Performing sentiment analysis on the target speech through a preset sentiment analysis module to obtain sentiment analysis features; Cross-domain fusion of the sentiment analysis feature and the pre-encoded facial attribute latent vector to generate a facial expression latent vector of the target object; Inputting the facial expression latent vector into the first face image generation module to generate an image, thereby obtaining a facial expression image of the target object; The facial expression image and the target mouth image are spliced ​​to obtain the target face image.

8. A face video inference device, characterized in that: The device comprises: An acquisition unit, used to acquire a target voice and a pre-encoded facial attribute latent vector for driving a target object, wherein the target voice includes a plurality of continuous time zones; An encoding unit, used for performing feature encoding on the target speech through a preset speech encoding module to obtain a target mouth shape latent vector; A fusion unit, configured to perform feature fusion on the face attribute latent vector and the target mouth shape latent vector through a pre-trained cross-domain fusion module to obtain a target latent vector; A generating unit, configured to input the target latent vector into a pre-trained first face image generating module for image generation to obtain a target face image of the target object; The inference unit is used to perform video inference according to the time zone sequence of the target speech and the multiple frames of the target face images corresponding to each time zone to obtain the target face video.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the face video inference method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the face video inference method described in any one of claims 1 to 7 is implemented.

Images