A facial expression capturing method and device, computer equipment and storage medium

Abstract

The present disclosure provides a facial expression capturing method and device, computer equipment and a storage medium, wherein the method comprises: obtaining face driving image data; performing first feature extraction on the face driving image data to obtain first expression features of the face driving image data; generating a virtual face image based on the first expression features and performing second feature extraction on the virtual face image to obtain second expression features of the virtual face image; and generating shape fusion BS coefficients based on the second expression features; the BS coefficients are used to input a three-dimensional game engine to generate a three-dimensional virtual face model. The present embodiment can automatically obtain BS coefficients based on face driving image data, so that a 3D virtual character model can be quickly driven based on the automatically obtained BS coefficients.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a method, apparatus, computer device, and storage medium for capturing facial expressions. Background Technology

[0002] In some activities, actors want to control 3D virtual character models through their facial expressions. Actors can drive relevant devices to capture their facial expressions, causing the 3D virtual character model to display the same expressions. To obtain a 3D virtual character model that can reproduce the same facial expressions as the actor, it is usually necessary to manually annotate the blend shape (BS) coefficients corresponding to the actor's facial expressions, and then use these BS coefficients to drive the 3D virtual character model. This manual annotation method is inefficient and affects the driving efficiency of the 3D virtual character model. Summary of the Invention

[0003] This disclosure provides at least one facial expression capture method, apparatus, computer device, and storage medium.

[0004] In a first aspect, embodiments of this disclosure provide a facial expression capture method, including:

[0005] Acquire face-driven image data;

[0006] The face-driven image data is subjected to a first feature extraction to obtain the first expression feature of the face-driven image data;

[0007] A virtual face image is generated based on the first expression feature, and a second feature is extracted from the virtual face image to obtain the second expression feature of the virtual face image;

[0008] Based on the second facial expression feature, shape fusion BS coefficients are generated; the BS coefficients are used to input into a 3D game engine to generate a 3D virtual face model.

[0009] In one optional implementation, the BS coefficients are generated by inputting the face-driven image data into a pre-trained face processing model.

[0010] The face processing model includes a first sub-face processing model and a second sub-face processing model; the first sub-face processing model is used to output the second expression feature based on the face-driven image data, and the second sub-face processing model is used to obtain the BS coefficient based on the second expression feature;

[0011] The first sub-face processing model includes an encoder, a first decoder, and a second decoder. The encoder is used to extract features from the image to obtain expression features. The first decoder is used to decode the expression features to obtain a virtual face generation image. The second decoder is used to decode the expression features to obtain a face-driven data generation image.

[0012] In one optional implementation, the first sub-face processing model is trained through the following steps:

[0013] Acquire face-driven image data samples and virtual face image samples;

[0014] The face-driven image data sample is encoded by the encoder of the first sub-face processing model to obtain a first controller expression feature, and the first controller expression feature is input into the first decoder of the first sub-face processing model to obtain a first virtual face generated image; and the virtual face image sample is encoded by the encoder of the first sub-face processing model to obtain a first virtual expression feature, and the first virtual expression feature is input into the second decoder of the first sub-face processing model to obtain a first face-driven data generated image;

[0015] The first virtual face image is encoded by the encoder to obtain a second virtual expression feature; and the first face driving data image is encoded by the encoder to obtain a second controller expression feature.

[0016] Based on the first loss information between the first control personnel's facial expression features and the second virtual facial expression features, and the second loss information between the first virtual facial expression features and the second control personnel's facial expression features, the model parameter information of the first sub-face processing model is adjusted.

[0017] In an optional implementation, after encoding the face-driven image data sample using the encoder of the first sub-face processing model to obtain the first controller's facial expression features, the method further includes:

[0018] The facial expression features of the first controller are input into the second decoder of the first sub-face processing model to obtain the second face-driven data generated image;

[0019] Based on the first image information of the face-driven image data sample and the second image information of the image generated from the second face-driven data, the third loss information is determined;

[0020] After encoding the virtual face image sample using the encoder of the first sub-face processing model to obtain the first virtual expression feature, the method further includes:

[0021] The first virtual facial expression feature is input into the first decoder of the first sub-face processing model to obtain the second virtual face generated image;

[0022] Based on the third image information of the virtual face image sample and the fourth image information of the second virtual face generated image, the fourth loss information is determined;

[0023] The step of adjusting the model parameter information of the first sub-face processing model based on the first loss information between the first controller's facial expression features and the second virtual facial expression features, and the second loss information between the first virtual facial expression features and the second controller's facial expression features, includes:

[0024] Based on the first loss information between the first control person's facial expression features and the second virtual facial expression features, the second loss information between the first virtual facial expression features and the second control person's facial expression features, the third loss information, and the fourth loss information, the model parameter information of the first sub-face processing model is adjusted.

[0025] In one optional implementation, determining the third loss information based on the first image information of the face-driven image data sample and the second image information of the image generated from the second face-driven data includes:

[0026] Based on the first pixel value information of each pixel in the face-driven image data sample and the second pixel value information of each pixel in the image generated by the second face-driven data, the first image quality loss information is determined.

[0027] Based on the first brightness information of the face-driven image data sample and the second brightness information of the image generated from the second face-driven data, image brightness loss information is determined;

[0028] Based on the first contrast information of the face-driven image data sample and the second contrast information of the image generated from the second face-driven data, image contrast loss information is determined;

[0029] Based on the first structural information of the face-driven image data sample and the second structural information of the image generated from the second face-driven data, image structural loss information is determined;

[0030] The second image quality loss information is determined based on the image brightness loss information, the image contrast loss information, and the image structure loss information;

[0031] Based on the first image quality loss information and the second image quality loss information, the third loss information is determined.

[0032] In an optional implementation, after obtaining the first virtual face generated image and the first face-driven data generated image, the method further includes:

[0033] The face-driven image data samples and the first virtual face generated image are input into a pre-trained discriminator to obtain a first authenticity judgment result for the face-driven image data samples and a second authenticity judgment result for the first virtual face generated image; based on the first authenticity judgment result and the second authenticity judgment result, the model parameter information of the first sub-face processing model is adjusted until the second authenticity judgment result of the first virtual face generated image matches the first authenticity judgment result of the face-driven image data samples; and / or

[0034] The virtual face image sample and the image generated from the first face-driven data are input into a pre-trained discriminator to obtain a third authenticity judgment result for the virtual face image sample and a fourth authenticity judgment result for the image generated from the first face-driven data. Based on the third authenticity judgment result and the fourth authenticity judgment result, the model parameter information of the first sub-face processing model is adjusted until the fourth authenticity judgment result of the image generated from the first face-driven data matches the third authenticity judgment result of the virtual face image sample.

[0035] In one optional implementation, the second sub-face processing model is trained through the following steps:

[0036] The virtual face image sample is input into the encoder of the trained first sub-face processing model to obtain the third virtual expression feature;

[0037] The third virtual expression feature is input into the second sub-face processing model to obtain the predicted BS coefficients corresponding to the virtual face image sample;

[0038] Based on the predicted BS coefficients and the known BS ​​coefficients corresponding to the virtual face image samples, the fifth loss information is determined.

[0039] Based on the fifth loss information, the model parameter information of the second sub-face processing model is adjusted.

[0040] In one optional implementation, acquiring face-driven image data samples and virtual face image samples includes:

[0041] Acquire raw face-driven image data and raw virtual face image;

[0042] Augmentation processing is performed on the original face-driven image data and the original virtual face image respectively to obtain augmented face-driven image data samples and augmented virtual face image samples.

[0043] The augmented face-driven image data and the augmented virtual face image are segmented to obtain face-driven image data samples containing the first face region and virtual face image samples containing the second face region.

[0044] In one optional implementation, the step of segmenting the augmented face-driven image data and the augmented virtual face image to obtain a face-driven image data sample containing a first face region and a virtual face image sample containing a second face region includes:

[0045] Face detection, face key point detection and face registration are performed sequentially on the augmented face-driven image data and the augmented virtual face image to determine the first face region of the augmented face-driven image data and the second face region of the augmented virtual face image.

[0046] Based on the first face region, the augmented face-driven image data is segmented to obtain a face-driven image data sample containing the first face region. Based on the second face region, the augmented virtual face image is segmented to obtain a virtual face image sample containing the second face region.

[0047] Secondly, embodiments of this disclosure also provide a facial expression capture device, comprising:

[0048] The first acquisition module is used to acquire face-driven image data;

[0049] The first extraction module is used to perform a first feature extraction on the face-driven image data to obtain the first expression feature of the face-driven image data;

[0050] The second extraction module is used to generate a virtual face image based on the first expression feature, and to extract a second feature from the virtual face image to obtain the second expression feature of the virtual face image.

[0051] The generation module is used to generate shape fusion BS coefficients based on the second facial expression features; the BS coefficients are used to input into a 3D game engine to generate a 3D virtual face model.

[0052] Thirdly, embodiments of this disclosure also provide a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0053] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible implementation thereof.

[0054] The facial expression capture method provided in this embodiment extracts first features from the acquired face-driven image data to obtain first expression features; then, a virtual face image is generated based on the first expression features, and second features are extracted from the virtual face image to obtain second expression features; finally, Black-Scholes coefficients are generated based on the second expression features. Through the above process, Black-Scholes coefficients can be automatically obtained from the face-driven image data, thereby enabling the rapid driving of 3D virtual character models based on the automatically obtained Black-Scholes coefficients.

[0055] Furthermore, in the process of training the face processing model, this embodiment adjusts the model parameter information of the first sub-face processing model by using expression feature loss information in different domains, namely: the first loss information between the first controller expression feature of the face-driven image data sample and the second virtual expression feature of the first virtual face generated image, and the second loss information between the first virtual expression feature of the virtual face image sample and the second controller expression feature of the first face-driven data generated image. This allows the encoder in the trained first sub-face processing model to have a better ability to encode expression features across domains. By adjusting the model parameter information of the first sub-face processing model by using image loss information in different domains, namely: the third loss information between the first image information of the face-driven image data sample and the second image information of the second face-driven data generated image, and the fourth loss information between the third image information of the virtual face image sample and the fourth image information of the second virtual face generated image, this allows the image output by the decoder in the trained first sub-face processing model to be more similar to the input image. Through the above process, the face processing model can be trained without labeling the face-driven image data with BS coefficients. This allows the first sub-face processing model to output expression features, and the second sub-face processing model to automatically obtain relatively accurate BS coefficients based on these expression features.

[0056] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0058] Figure 1 A flowchart of a facial expression capture method provided by an embodiment of this disclosure is shown;

[0059] Figure 2 A flowchart of another facial expression capture method provided by an embodiment of this disclosure is shown;

[0060] Figure 3 A flowchart illustrating the training process of the first sub-face processing model provided in an embodiment of this disclosure is shown.

[0061] Figure 4 A schematic diagram of the process for obtaining training samples provided in an embodiment of this disclosure is shown;

[0062] Figure 5 A training flowchart of another first sub-face processing model provided in an embodiment of this disclosure is shown;

[0063] Figure 6 A schematic diagram of the process for training a first sub-face processing model using a discriminator, as provided in an embodiment of this disclosure, is shown.

[0064] Figure 7 A flowchart illustrating the training process of the second sub-face processing model provided in an embodiment of this disclosure is shown.

[0065] Figure 8 A training flowchart of another second sub-face processing model provided in an embodiment of this disclosure is shown;

[0066] Figure 9 A schematic diagram of the structure of a facial expression capture device provided in an embodiment of this disclosure is shown;

[0067] Figure 10 A schematic diagram of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0069] In practice, to obtain a 3D virtual character model that can reproduce the same facial expressions as an actor, after the actor's facial image is acquired by the relevant device, the Black-Scholes coefficients corresponding to the actor's facial image need to be manually annotated. Then, the 3D virtual character model is driven based on the actor's facial image and the corresponding Black-Scholes coefficients. This manual annotation method is inefficient and affects the driving efficiency of the 3D virtual character model.

[0070] Based on this, this disclosure provides a facial expression capture method. First, a first feature extraction is performed on the acquired face-driven image data to obtain first expression features. Then, a virtual face image is generated based on the first expression features, and second feature extraction is performed on the virtual face image to obtain second expression features. Finally, based on the second expression features, Black-Scholes (BS) coefficients are generated. Through the above process, BS coefficients can be automatically obtained from face-driven image data, thereby enabling rapid driving of 3D virtual character models based on the automatically obtained BS coefficients.

[0071] The deficiencies of the above solutions and the proposed solutions are the result of the inventors' practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventors' contributions to this disclosure.

[0072] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0073] To facilitate understanding of this embodiment, a facial expression capture method disclosed in this disclosure will first be described in detail. The facial expression capture method provided in this disclosure is generally executed by a computer device with certain computing power.

[0074] The facial expression capture method provided in this disclosure is illustrated below, taking the server as the executing entity as an example.

[0075] See Figure 1 The diagram shows a flowchart of a facial expression capture method provided in an embodiment of this disclosure. The method includes steps S101 to S104, wherein:

[0076] S101: Acquire face-driven image data.

[0077] S102: Perform a first feature extraction on the face-driven image data to obtain the first expression feature of the face-driven image data.

[0078] S103: Generate a virtual face image based on the first expression feature, and extract the second feature from the virtual face image to obtain the second expression feature of the virtual face image.

[0079] S104: Based on the second facial expression feature, generate shape fusion BS coefficients; the BS coefficients are used to input into a 3D game engine to generate a 3D virtual face model.

[0080] In this embodiment of the disclosure, face-driven image data can refer to an image of the controller's face. This image data can be images of various facial expressions, such as smiling, angry, fear, etc. Here, an image containing the controller's face can be obtained by image acquisition, for example, by taking a photo or recording a video. It should be noted that the process of acquiring face-driven image data can be triggered by the controller themselves or executed after authorization by the controller.

[0081] The acquired image containing the operator's face may be a raw image containing other parts besides the face (such as the neck, hair, etc.). These other parts in the raw image may affect the process of capturing the operator's facial expressions; for example, they may affect the extraction of facial features. Therefore, after acquiring the raw image, it can be preprocessed to obtain face-driven image data containing only the operator's face.

[0082] In one implementation of preprocessing the acquired raw images, face detection, facial landmark detection, and face registration are performed sequentially to determine the facial regions within the raw images. Then, based on these facial regions, the raw images are segmented to obtain face-driven image data containing only the controller's face. This preprocessing can, to some extent, eliminate the influence of other parts of the raw image on capturing the controller's facial expressions.

[0083] After extracting the first feature from the face-driven image data, the resulting first expression feature can be an expression feature in the Red Green Blue (RGB) color gamut. After obtaining the first expression feature from the face-driven image data, this feature can be loaded onto a virtual face to generate a virtual face image with the same facial expression as the face-driven image data.

[0084] After extracting the second feature from the virtual face image, the resulting second expression feature can be an expression feature in the field of computer graphics (CG).

[0085] Shape blending (BS) coefficients are used to input 3D game engines to generate 3D virtual face models. In other words, inputting BS coefficients into a 3D game engine generates a 3D virtual face model. The facial expressions of the 3D virtual face model generated based on BS coefficients are consistent with those of a virtual face image, and consequently, with those of the facial expressions in the face-driven image data. This enables the capture of facial expressions from face-driven image data, which can then be applied to the 3D virtual face model.

[0086] In one implementation, the facial expression capture process described above can be based on a pre-trained face processing model. Specifically, the generation of BS coefficients is performed after inputting the face-driven image data into the pre-trained face processing model. In other words, by inputting the acquired face-driven image data into the pre-trained face processing model, BS coefficients can be obtained.

[0087] The face processing model may include a first sub-face processing model and a second sub-face processing model. The first sub-face processing model outputs a second expression feature based on face-driven image data; the second sub-face processing model obtains BS coefficients based on the second expression feature. For example, the first sub-face processing model can be an autoencoder structure, and the second sub-face processing model can be a deep neural network (DNN).

[0088] Furthermore, the first sub-face processing model may include an encoder, a first decoder, and a second decoder; wherein, the encoder is used to extract features from the image to obtain expression features; the first decoder is used to decode the expression features to obtain a virtual face generation image; and the second decoder is used to decode the expression features to obtain a face-driven data generation image.

[0089] In specific implementation, see Figure 2 In the flowchart of another facial expression capture method shown, after acquiring the original image containing the face of the controller, the original image is processed by facial key point detection and face registration to obtain face-driven image data. Then, the face-driven image data is input into the encoder in the first sub-face processing model to extract the first feature and obtain the first expression feature of the face-driven image data. Then, the first decoder decodes the first expression feature to obtain a virtual face image. Next, the encoder extracts the second feature of the virtual face image to obtain the second expression feature of the virtual face image. Finally, the second sub-face processing model DNN obtains the BS coefficients based on the second expression feature.

[0090] The training processes of the first sub-face processing model and the second sub-face processing model in the above face processing model are not carried out simultaneously. In specific implementation, the training process of the second sub-face processing model can be carried out after the training of the first sub-face processing model is completed.

[0091] The training process of the first and second sub-face processing models will be described below, following the order in which they were trained.

[0092] The training process of the first sub-face processing model will be introduced below. See [link / reference] Figure 3 The flowchart shown illustrates the training process of the first sub-face processing model. The first sub-face processing model is trained through the following steps:

[0093] S301: Acquire face-driven image data samples and virtual face image samples.

[0094] In this embodiment of the disclosure, the face-driven image data sample refers to an image sample containing the face of the controller. The face-driven image data sample can be an image sample under various facial expressions, such as image samples under various expressions such as smiling, anger, and fear.

[0095] Here, image samples containing the face of the control personnel can be obtained by image acquisition, such as by taking photos or videos. It should be noted that the process of acquiring face-driven image data can be initiated by the control personnel themselves or by authorization from them.

[0096] Virtual face image samples refer to image samples containing virtual faces. These samples can be images of various facial expressions, such as smiling, angry, and fear. Here, multiple different Black-Scholes coefficients can be pre-generated, and then these coefficients are used to generate image samples containing the virtual object's face.

[0097] The acquired image samples containing the controller's face and the generated image samples containing the virtual object's face may be original face images containing other parts besides the face (such as the neck, hair, etc.), i.e., original face-driven image data and original virtual face images. Other parts in the original face-driven image data and original virtual face images may have a certain impact on the training process of the first sub-face processing model, for example, it may affect the extraction of facial expression features. Therefore, after acquiring the original face-driven image data and original virtual face images, they can be preprocessed separately to obtain face-driven image data samples containing only the controller's face and image samples containing only the virtual object's face.

[0098] In one implementation of the preprocessing process for the acquired original face-driven image data and original virtual face image, augmentation processing is first performed on both the original face-driven image data and the original virtual face image to obtain augmented face-driven image data samples and augmented virtual face image samples. Augmentation processing can add information or transform image features to the original image, selectively highlighting or suppressing certain features in the original image. This increases the number of image samples, thereby improving the training accuracy of the first sub-face processing model and making the second expression features obtained by the trained first sub-face processing model more accurate.

[0099] Next, the augmented face-driven image data and the augmented virtual face image can be segmented separately to obtain face-driven image data samples containing the first face region and virtual face image samples containing the second face region. The first face region is the face region of the controller, and the second face region is the face region of the virtual object.

[0100] In a further embodiment, during the segmentation process of the augmented face-driven image data and the augmented virtual face image, such as Figure 4 The flowchart illustrating the process of obtaining training samples shows that face detection, facial landmark detection, and face registration are performed sequentially on the augmented face-driven image data and the augmented virtual face image to determine the first face region of the augmented face-driven image data and the second face region of the augmented virtual face image. Then, based on the first face region, the augmented face-driven image data is segmented to obtain face-driven image data samples containing the first face region, and based on the second face region, the augmented virtual face image is segmented to obtain virtual face image samples containing the second face region. Furthermore, masking can be performed on the face-driven image data samples containing the first face region and the virtual face image samples containing the second face region to obtain a first mask image and a second mask image. This preprocessing of the original face-driven image data and the original virtual face image can, to some extent, eliminate the influence of other parts of the original face-driven image data and the original virtual face image on the training process of the first sub-face processing model.

[0101] S302: The face-driven image data sample is encoded by the encoder of the first sub-face processing model to obtain a first controller expression feature, and the first controller expression feature is input into the first decoder of the first sub-face processing model to obtain a first virtual face generated image; and the virtual face image sample is encoded by the encoder of the first sub-face processing model to obtain a first virtual expression feature, and the first virtual expression feature is input into the second decoder of the first sub-face processing model to obtain a first face-driven data generated image.

[0102] The facial expression features of the first controller can be facial expression features in the RGB domain. After inputting the facial expression features of the first controller into the first decoder of the first sub-face processing model, the first decoder can obtain a first virtual face generated image based on the facial expression features of the first controller. The generated first virtual face generated image has the same facial expression as the face-driven image data sample.

[0103] The first virtual expression feature is also an expression feature in the RGB domain. After inputting the first virtual expression feature into the second decoder of the first sub-face processing model, the second decoder can obtain a first face-driven data generated image based on the first virtual expression feature. The generated first face-driven data generated image has the same facial expression as the virtual face image sample.

[0104] S303: Encode the first virtual face image using the encoder to obtain a second virtual expression feature; and encode the first face driving data image using the encoder to obtain a second controller expression feature.

[0105] The second virtual facial expression feature and the second controller's facial expression feature can both be facial expression features under the CG domain.

[0106] Based on facial expression features in the RGB and CG domains, loss information for facial expression features in different domains can be obtained. Based on this loss information, the model parameters of the first sub-face processing model can be adjusted. By using this loss information from different domains, the encoder in the trained first sub-face processing model can better encode facial expression features across domains.

[0107] S304: Based on the first loss information between the first control personnel's facial expression features and the second virtual facial expression features, and the second loss information between the first virtual facial expression features and the second control personnel's facial expression features, adjust the model parameter information of the first sub-face processing model.

[0108] As mentioned above, the generated first virtual face image and the face-driven image data sample have the same facial expression. That is, the first controller's facial expression feature and the second virtual facial expression feature correspond to the same facial expression, except that the first controller's facial expression feature and the second virtual facial expression feature are facial expression features in the RGB domain and CG domain, respectively. Therefore, the first loss information can be determined based on the first controller's facial expression feature and the second virtual facial expression feature. Similarly, the second loss information can be determined based on the first virtual facial expression feature and the second controller's facial expression feature.

[0109] Based on the first loss information and the second loss information, the model parameters of the first sub-face processing model are adjusted, and the first sub-face processing model is trained to obtain the trained first sub-face processing model.

[0110] The first and second loss information can be calculated using the Cycle Consistency Loss (CCL) loss function formula, respectively:

[0111] In calculating the first loss information, x i Enc(x) represents a face-driven image data sample. i ) is the first controller's facial expression features in the face-driven image data samples, Dec j (Enc(x iEnc(Dec) is a first virtual face image generated based on the facial expression features of the first controller. j (Enc(x i The second virtual expression feature is generated from the image of the first virtual face. When calculating the second loss information, x... i Enc(x) represents a virtual face image sample. i ) represents the first virtual expression feature of the virtual face image sample, Dec j (Enc(x i Enc(Dec) generates an image based on the first face-driven data generated from the first virtual expression features. j (Enc(x i The second controller's facial expression features are used to generate an image based on the first face-driven data.

[0112] To improve the image quality of the generated image, in one implementation, the model parameters of the first sub-face processing model can be adjusted based on the image quality loss information between the generated face image in the CG domain and the face image in the RGB domain.

[0113] Specifically, after encoding the face-driven image data samples into the first controller's facial expression features using the encoder of the first sub-face processing model, the first controller's facial expression features can be input into the second decoder of the first sub-face processing model to obtain the second face-driven data generated image. Here, the second face-driven data generated image is a face image in the CG domain. The second face-driven data generated image has the same facial expression as the face-driven image data samples.

[0114] Then, based on the first image information of the face-driven image data sample and the second image information of the face-driven image generated from the second face-driven data, the third loss information is determined. The third loss information is the image quality loss information between the image information of the second face-driven image generated in the CG domain and the image information of the face-driven image data in the RGB domain.

[0115] The first virtual expression feature is input into the first decoder of the first sub-face processing model to obtain a second virtual face image. Here, the second virtual face image is a face image in the CG domain. The second virtual face image has the same facial expression as the virtual face image sample.

[0116] Then, based on the third image information of the virtual face image sample and the fourth image information of the second virtual face generated image, the fourth loss information is determined. The fourth loss information is the image quality loss information between the image information of the second virtual face generated image in the CG domain and the image information of the virtual face image in the RGB domain.

[0117] Finally, based on the first loss information between the first control person's facial expression features and the second virtual facial expression features, the second loss information between the first virtual facial expression features and the second control person's facial expression features, the third loss information, and the fourth loss information, the model parameter information of the first sub-face processing model is adjusted.

[0118] In the above embodiments, the image information may include pixel value information of each pixel in the image, brightness information of the image, contrast information of the image, structural information of the image, etc.

[0119] In specific implementation, in the process of determining the third or fourth loss information, the first image quality loss information can be determined based on the pixel value information of each pixel in the image, and the third or fourth loss information can be determined based on the first image quality loss information; or the second image quality loss information can be determined based on the brightness information, contrast information, and structure information of the image, and the third or fourth loss information can be determined based on the second image quality loss information; or the third or fourth loss information can be determined based on both the first and second image quality loss information.

[0120] It should be noted that in other implementations, image quality loss information can also be determined based on other image information, which will not be described in detail here.

[0121] In specific implementation, see Figure 5 The training flowchart of another first sub-face processing model is shown. The face-driven image data sample is input into the encoder of the first sub-face processing model to encode the first controller's facial expression features. The first controller's facial expression features are input into the first decoder of the first sub-face processing model to obtain the first virtual face generated image. The first controller's facial expression features are input into the second decoder of the first sub-face processing model to obtain the second face-driven data generated image.

[0122] The virtual face image sample is input into the encoder of the first sub-face processing model to obtain the first virtual expression feature. The first virtual expression feature is input into the second decoder of the first sub-face processing model to obtain the first face-driven data generated image. The first virtual expression feature is input into the first decoder of the first sub-face processing model to obtain the second virtual face generated image.

[0123] Then, the first virtual face image is input into the encoder to obtain the second virtual expression feature; and the first face driving data image is input into the encoder to obtain the second controller expression feature.

[0124] Finally, based on the facial expression features of the first controller and the second virtual facial expression features, the first loss information is determined; based on the first virtual facial expression features and the second controller facial expression features, the second loss information is determined; based on the first image information of the face-driven image data samples and the second image information of the image generated from the second face-driven data, the third loss information is determined; based on the third image information of the virtual face image samples and the fourth image information of the image generated from the second virtual face, the fourth loss information is determined.

[0125] Then, the model parameters of the first sub-face processing model can be adjusted based on the first loss information, the second loss information, the third loss information, and the fourth loss information.

[0126] The following section will use the determination of third loss information as an example to introduce the process of determining third loss information based on first image quality loss information and second image quality loss information.

[0127] Here, the first image quality loss information can be determined based on the first pixel value information of each pixel in the face-driven image data sample and the second pixel value information of each pixel in the image generated from the second face-driven data.

[0128] In a specific implementation, the face-driven image data sample and the second face-driven data generated image can be pre-masked to obtain the mask image corresponding to the face-driven image data sample and the mask image corresponding to the second face-driven data generated image. Then, based on the first pixel value information of each pixel in the mask image corresponding to the face-driven image data sample and the second pixel value information of each pixel in the mask image corresponding to the second face-driven data generated image, the first image quality loss is determined.

[0129] Furthermore, image brightness loss information is determined based on the first brightness information of the face-driven image data sample and the second brightness information of the image generated from the second face-driven data; image contrast loss information is determined based on the first contrast information of the face-driven image data sample and the second contrast information of the image generated from the second face-driven data; and image structure loss information is determined based on the first structural information of the face-driven image data sample and the second structural information of the image generated from the second face-driven data.

[0130] Here, we can first obtain the mask image corresponding to the face-driven image data sample and the mask image corresponding to the second face-driven data generated image, and then determine the image brightness loss information, image contrast loss information and image structure loss information respectively.

[0131] Then, based on the image brightness loss information, image contrast loss information, and image structure loss information, the second image quality loss information is determined.

[0132] Finally, based on the first image quality loss information and the second image quality loss information, the third loss information is determined. Here, the first image quality loss information and the second image quality loss information can be weighted and summed to obtain the third loss information.

[0133] Here, the third loss information can be Where, x i When representing face-driven image data samples, f(x) i ) represents the first pixel value information of each pixel in the face-driven image data sample, y i This represents the second pixel value information of each pixel in the image generated by the second face-driven data; x i The virtual face image sample is represented by f(x). i ) represents the first pixel value information of each pixel point in the virtual face image sample, y i This represents the second pixel value information of each pixel in the second virtual face generated image.

[0134] The process for determining the fourth loss information is similar to that for determining the third loss information, and will not be repeated here.

[0135] Here, the fourth loss information can be calculated using the Structural Similarity Index Measure (SSIM). Specifically, the fourth loss information is L. SSIM (x,y)=[l(x,y) α ·c(x,y) β ·s(x,y) γ ], α, β, γ are greater than 0, and are used to adjust the importance of the three modules.

[0136] in This represents the image brightness loss information. C1 is a constant to prevent system instability caused by the sum of squares in the denominator approaching 0. C1 = (K1L) 2 L is the number of gray levels in the image, typically L = 255, and K1 << 1. When x is a sample of face-driven image data and y is an image generated from second face-driven data, μ xFor the brightness information of face-driven image data samples, μ y The brightness information of the image generated from the second face-driven data. When x is a virtual face image sample and y is the generated image of the second virtual face, μ x For the brightness information of virtual face image samples, μ y Brightness information for generating an image of the second virtual face.

[0137] This represents the image contrast loss information, with the constant C2 = (K2L). 2 And K2 << 1. When x is a sample of face-driven image data and y is an image generated from second face-driven data, σ x For the contrast information of face-driven image data samples, σ y Contrast information for the image generated from the second face-driven data. When x is a virtual face image sample and y is the generated image of the second virtual face, σ x For the contrast information of virtual face image samples, σ y Contrast information for generating an image of the second virtual face.

[0138] Represents image structural loss information, and

[0139] To improve the quality of the generated images, a discriminator can be used to train the first sub-face processing model simultaneously. The discriminator can be pre-trained. The discriminator and the first sub-face processing model form an adversarial network. The discriminator's judgment results on the generated images are used to optimize the first sub-face processing model, making the first virtual face generated by the first sub-face processing model more similar to the face-driven image data samples, and making the first face-driven data generated images more similar to the virtual image samples.

[0140] In one implementation, see Figure 6 The flowchart illustrating the training process of the first sub-face processing model using a discriminator shows that face-driven image data samples and first virtual face generated images can be input into a pre-trained discriminator to obtain a first true / false discrimination result for the face-driven image data samples and a second true / false discrimination result for the first virtual face generated images. Based on the first and second true / false discrimination results, the model parameters of the first sub-face processing model are adjusted until the second true / false discrimination result of the first virtual face generated images matches the first true / false discrimination result of the face-driven image data samples.

[0141] A pre-trained discriminator can determine whether the face-driven image data sample is real or not; this is the first real / false judgment result. At the beginning of training the first sub-face processing model, because the similarity between the first virtual face image generated by the first sub-face processing model and the face-driven image data sample is small, the discriminator may obtain a real / false judgment result where the first virtual face image is not real; this is the second real / false judgment result. At this point, the model parameters of the first sub-face processing model can be adjusted based on the first and second real / false judgment results.

[0142] Then, the first virtual face image generated by the first sub-face processing model after adjusting the model parameter information is input into the discriminator to obtain the second authenticity judgment result of the first virtual face image again. If the second authenticity judgment result still indicates that the first virtual face image is not real, the model parameter information of the first sub-face processing model is adjusted until the second authenticity judgment result of the first virtual face image matches the first authenticity judgment result of the face-driven image data sample.

[0143] In one implementation, the first and second true / false discrimination results can be represented by probability values. For example, a true / false discrimination result can be represented by 1, and a false discrimination result can be represented by 0. When the difference between the second true / false discrimination result of the first virtual face generated image and the first true / false discrimination result of the face-driven image data sample is less than a set threshold, it can be considered that the second true / false discrimination result of the first virtual face generated image matches the first true / false discrimination result of the face-driven image data sample, and training can then end.

[0144] And / or, input the virtual face image sample and the image generated by the first face-driven data into a pre-trained discriminator to obtain the third authenticity judgment result of the virtual face image sample and the fourth authenticity judgment result of the image generated by the first face-driven data; based on the third authenticity judgment result and the fourth authenticity judgment result, adjust the model parameter information of the first sub-face processing model until the fourth authenticity judgment result of the image generated by the first face-driven data matches the third authenticity judgment result of the virtual face image sample.

[0145] Here, the model parameter information of the first sub-face processing model can be adjusted based on the third authenticity judgment result of the virtual face image sample and the fourth authenticity judgment result of the image generated by the first face-driven data, and will not be described in detail here.

[0146] The training process of the first sub-face processing model has been described above. The training process of the second sub-face processing model will be described below. (See also...) Figure 7The flowchart shown illustrates the training process for the second sub-face processing model. The second sub-face processing model is trained through the following steps:

[0147] S701: Input the virtual face image sample into the encoder of the trained first sub-face processing model to obtain the third virtual expression feature;

[0148] S702: Input the third virtual expression feature into the second sub-face processing model to obtain the predicted BS coefficients corresponding to the virtual face image sample;

[0149] S703: Determine the fifth loss information based on the predicted BS coefficients and the known BS ​​coefficients corresponding to the virtual face image samples;

[0150] S704: Based on the fifth loss information, adjust the model parameter information of the second sub-face processing model.

[0151] Here, the first sub-face processing model is trained. The encoder of the trained first sub-face processing model encodes the virtual face image samples, resulting in more accurate third virtual expression features.

[0152] Here, the second sub-face processing model can be a DNN, which can predict the BS coefficients corresponding to the virtual face image samples based on the third virtual expression features.

[0153] The known BS ​​coefficients corresponding to the virtual face image samples refer to the BS coefficients used to generate the virtual face image samples. Here, the mean square error (MSE) loss, i.e., the fifth loss information, can be calculated based on the predicted BS coefficients and the known BS ​​coefficients corresponding to the virtual face image samples.

[0154] Based on the fifth loss information, the model parameters of the second sub-face processing model are adjusted to obtain the trained second sub-face processing model.

[0155] In specific implementation, see Figure 8 The training flowchart of another second sub-face processing model is shown. The virtual face image sample is input into the encoder of the trained first sub-face processing model to obtain the third virtual expression feature. The third virtual expression feature is input into the DNN to obtain the predicted BS coefficients corresponding to the virtual face image sample. Then, based on the predicted BS coefficients and the known BS ​​coefficients corresponding to the virtual face image sample, the fifth loss information is determined.

[0156] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0157] Based on the same inventive concept, this disclosure also provides a facial expression capture device corresponding to the facial expression capture method. Since the principle of the device in this disclosure is similar to that of the facial expression capture method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0158] Reference Figure 9 The diagram shown is a structural schematic of a facial expression capture device provided in an embodiment of this disclosure. The device includes: a first acquisition module 901, a first extraction module 902, a second extraction module 903, and a generation module 904; wherein,

[0159] The first acquisition module 901 is used to acquire face-driven image data;

[0160] The first extraction module 902 is used to perform a first feature extraction on the face-driven image data to obtain the first expression feature of the face-driven image data;

[0161] The second extraction module 903 is used to generate a virtual face image based on the first expression feature, and to extract a second feature from the virtual face image to obtain the second expression feature of the virtual face image.

[0162] The generation module 904 is used to generate shape fusion BS coefficients based on the second facial expression features; the BS coefficients are used to input into a 3D game engine to generate a 3D virtual face model.

[0163] In one optional implementation, the BS coefficients are generated by inputting the face-driven image data into a pre-trained face processing model.

[0164] The face processing model includes a first sub-face processing model and a second sub-face processing model; the first sub-face processing model is used to output the second expression feature based on the face-driven image data, and the second sub-face processing model is used to obtain the BS coefficient based on the second expression feature;

[0165] The first sub-face processing model includes an encoder, a first decoder, and a second decoder. The encoder is used to extract features from the image to obtain expression features. The first decoder is used to decode the expression features to obtain a virtual face generation image. The second decoder is used to decode the expression features to obtain a face-driven data generation image.

[0166] In one optional embodiment, the apparatus further includes:

[0167] The second acquisition module is used to acquire face-driven image data samples and virtual face image samples;

[0168] The first input module is configured to encode the face-driven image data sample through the encoder of the first sub-face processing model to obtain a first controller expression feature, input the first controller expression feature into the first decoder of the first sub-face processing model to obtain a first virtual face generated image; and to encode the virtual face image sample through the encoder of the first sub-face processing model to obtain a first virtual expression feature, input the first virtual expression feature into the second decoder of the first sub-face processing model to obtain a first face-driven data generated image;

[0169] The encoding module is used to encode the first virtual face generated image through the encoder to obtain a second virtual expression feature; and to encode the first face driving data generated image through the encoder to obtain a second controller expression feature;

[0170] The first adjustment module is used to adjust the model parameter information of the first sub-face processing model based on the first loss information between the first control personnel's facial expression features and the second virtual facial expression features, and the second loss information between the first virtual facial expression features and the second control personnel's facial expression features.

[0171] In an optional implementation, after the face-driven image data sample is encoded by the encoder of the first sub-face processing model to obtain the first controller's facial expression features, the device further includes:

[0172] The second input module is used to input the facial expression features of the first controller into the second decoder of the first sub-face processing model to obtain the second face-driven data generated image;

[0173] The first determining module is used to determine the third loss information based on the first image information of the face-driven image data sample and the second image information of the image generated from the second face-driven data.

[0174] After encoding the virtual face image sample into a first virtual expression feature using the encoder of the first sub-face processing model, the device further includes:

[0175] The third input module is used to input the first virtual expression features into the first decoder of the first sub-face processing model to obtain the second virtual face generation image;

[0176] The second determining module is used to determine the fourth loss information based on the third image information of the virtual face image sample and the fourth image information of the second virtual face generated image;

[0177] The first adjustment module is specifically used for:

[0178] Based on the first loss information between the first control person's facial expression features and the second virtual facial expression features, the second loss information between the first virtual facial expression features and the second control person's facial expression features, the third loss information, and the fourth loss information, the model parameter information of the first sub-face processing model is adjusted.

[0179] In one optional implementation, the first determining module is specifically used for:

[0180] Based on the first pixel value information of each pixel in the face-driven image data sample and the second pixel value information of each pixel in the image generated by the second face-driven data, the first image quality loss information is determined.

[0181] Based on the first brightness information of the face-driven image data sample and the second brightness information of the image generated from the second face-driven data, image brightness loss information is determined;

[0182] Based on the first contrast information of the face-driven image data sample and the second contrast information of the image generated from the second face-driven data, image contrast loss information is determined;

[0183] Based on the first structural information of the face-driven image data sample and the second structural information of the image generated from the second face-driven data, image structural loss information is determined;

[0184] The second image quality loss information is determined based on the image brightness loss information, the image contrast loss information, and the image structure loss information;

[0185] Based on the first image quality loss information and the second image quality loss information, the third loss information is determined.

[0186] In an optional implementation, after obtaining the first virtual face generated image and the first face-driven data generated image, the apparatus further includes:

[0187] The fourth input module is used to input the face-driven image data sample and the first virtual face generated image into a pre-trained discriminator to obtain a first authenticity judgment result of the face-driven image data sample and a second authenticity judgment result of the first virtual face generated image; based on the first authenticity judgment result and the second authenticity judgment result, adjust the model parameter information of the first sub-face processing model until the second authenticity judgment result of the first virtual face generated image matches the first authenticity judgment result of the face-driven image data sample; and / or

[0188] The virtual face image sample and the image generated from the first face-driven data are input into a pre-trained discriminator to obtain a third authenticity judgment result for the virtual face image sample and a fourth authenticity judgment result for the image generated from the first face-driven data. Based on the third authenticity judgment result and the fourth authenticity judgment result, the model parameter information of the first sub-face processing model is adjusted until the fourth authenticity judgment result of the image generated from the first face-driven data matches the third authenticity judgment result of the virtual face image sample.

[0189] In one optional embodiment, the apparatus further includes:

[0190] The fifth input module is used to input the virtual face image sample into the encoder of the trained first sub-face processing model to obtain the third virtual expression feature;

[0191] The sixth input module is used to input the third virtual expression feature into the second sub-face processing model to obtain the predicted BS coefficients corresponding to the virtual face image sample;

[0192] The third determining module is used to determine the fifth loss information based on the predicted BS coefficients and the known BS ​​coefficients corresponding to the virtual face image samples;

[0193] The second adjustment module is used to adjust the model parameter information of the second sub-face processing model based on the fifth loss information.

[0194] In one optional implementation, the second acquisition module is specifically used for:

[0195] Acquire raw face-driven image data and raw virtual face image;

[0196] Augmentation processing is performed on the original face-driven image data and the original virtual face image respectively to obtain augmented face-driven image data samples and augmented virtual face image samples.

[0197] The augmented face-driven image data and the augmented virtual face image are segmented to obtain face-driven image data samples containing the first face region and virtual face image samples containing the second face region.

[0198] In one optional implementation, the second acquisition module is specifically used for:

[0199] Face detection, face key point detection and face registration are performed sequentially on the augmented face-driven image data and the augmented virtual face image to determine the first face region of the augmented face-driven image data and the second face region of the augmented virtual face image.

[0200] Based on the first face region, the augmented face-driven image data is segmented to obtain a face-driven image data sample containing the first face region. Based on the second face region, the augmented virtual face image is segmented to obtain a virtual face image sample containing the second face region.

[0201] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0202] Based on the same technical concept, this disclosure also provides a computer device. (See also...) Figure 10 The diagram shows the structure of a computer device 1000 provided in this embodiment of the present disclosure, including a processor 1001, a memory 1002, and a bus 1003. The memory 1002 stores execution instructions and includes main memory 10021 and external memory 10022. The main memory 10021, also called internal memory, is used to temporarily store computational data in the processor 1001 and data exchanged with external memory 10022 such as a hard disk. The processor 1001 exchanges data with the external memory 10022 through the main memory 10021. When the computer device 1000 is running, the processor 1001 and the memory 1002 communicate through the bus 1003, causing the processor 1001 to execute the following instructions:

[0203] Acquire face-driven image data;

[0204] The face-driven image data is subjected to a first feature extraction to obtain the first expression feature of the face-driven image data;

[0205] A virtual face image is generated based on the first expression feature, and a second feature is extracted from the virtual face image to obtain the second expression feature of the virtual face image;

[0206] Based on the second facial expression feature, shape fusion BS coefficients are generated; the BS coefficients are used to input into a 3D game engine to generate a 3D virtual face model.

[0207] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the facial expression capture method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0208] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the facial expression capture method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0209] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0210] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0211] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0213] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0214] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for capturing facial expressions, characterized in that, include: Acquire face-driven image data; The face-driven image data is subjected to a first feature extraction to obtain the first expression feature of the face-driven image data; A virtual face image is generated based on the first expression feature, and a second feature is extracted from the virtual face image to obtain the second expression feature of the virtual face image; Based on the second facial expression feature, shape fusion BS coefficients are generated; The BS coefficients are used as input to the 3D game engine to generate a 3D virtual face model. The BS coefficients are generated by inputting the face-driven image data into a pre-trained face processing model. The face processing model includes a first sub-face processing model and a second sub-face processing model; the first sub-face processing model is used to output the second expression feature based on the face-driven image data, and the second sub-face processing model is used to obtain the BS coefficient based on the second expression feature; The first sub-face processing model includes an encoder, a first decoder, and a second decoder. The encoder is used to extract features from the image to obtain expression features. The first decoder is used to decode the expression features to obtain a virtual face generation image. The second decoder is used to decode the expression features to obtain a face-driven data generation image. The first sub-face processing model is trained through the following steps: Acquire face-driven image data samples and virtual face image samples; The face-driven image data sample is encoded by the encoder of the first sub-face processing model to obtain a first controller expression feature, and the first controller expression feature is input into the first decoder of the first sub-face processing model to obtain a first virtual face generated image; and the virtual face image sample is encoded by the encoder of the first sub-face processing model to obtain a first virtual expression feature, and the first virtual expression feature is input into the second decoder of the first sub-face processing model to obtain a first face-driven data generated image; The first virtual face image is encoded by the encoder to obtain a second virtual expression feature; and the first face driving data image is encoded by the encoder to obtain a second controller expression feature. Based on the first loss information between the first control personnel's facial expression features and the second virtual facial expression features, and the second loss information between the first virtual facial expression features and the second control personnel's facial expression features, the model parameter information of the first sub-face processing model is adjusted.

2. The method according to claim 1, characterized in that, The acquisition of face-driven image data includes: The face image containing the face of the control personnel is processed by face detection, face key point detection and face registration to determine the face region in the face image; Based on the facial region, the facial image is segmented to obtain facial driving image data that only contains the face of the controller.

3. The method according to claim 1, characterized in that, After encoding the face-driven image data samples using the encoder of the first sub-face processing model to obtain the first controller's facial expression features, the method further includes: The facial expression features of the first controller are input into the second decoder of the first sub-face processing model to obtain the second face-driven data generated image; Based on the first image information of the face-driven image data sample and the second image information of the image generated from the second face-driven data, the third loss information is determined; After encoding the virtual face image sample using the encoder of the first sub-face processing model to obtain the first virtual expression feature, the method further includes: The first virtual facial expression feature is input into the first decoder of the first sub-face processing model to obtain the second virtual face generated image; Based on the third image information of the virtual face image sample and the fourth image information of the second virtual face generated image, the fourth loss information is determined; The step of adjusting the model parameter information of the first sub-face processing model based on the first loss information between the first controller's facial expression features and the second virtual facial expression features, and the second loss information between the first virtual facial expression features and the second controller's facial expression features, includes: Based on the first loss information between the first control person's facial expression features and the second virtual facial expression features, the second loss information between the first virtual facial expression features and the second control person's facial expression features, the third loss information, and the fourth loss information, the model parameter information of the first sub-face processing model is adjusted.

4. The method according to claim 3, characterized in that, The determination of third loss information based on the first image information of the face-driven image data sample and the second image information of the image generated from the second face-driven data includes: Based on the first pixel value information of each pixel in the face-driven image data sample and the second pixel value information of each pixel in the image generated by the second face-driven data, the first image quality loss information is determined. Based on the first brightness information of the face-driven image data sample and the second brightness information of the image generated from the second face-driven data, image brightness loss information is determined; Based on the first contrast information of the face-driven image data sample and the second contrast information of the image generated from the second face-driven data, image contrast loss information is determined; Based on the first structural information of the face-driven image data sample and the second structural information of the image generated from the second face-driven data, image structural loss information is determined; Based on the image brightness loss information, the image contrast loss information, and the image structure loss information, the second image quality loss information is determined. Based on the first image quality loss information and the second image quality loss information, the third loss information is determined.

5. The method according to claim 1, characterized in that, After obtaining the first virtual face generated image and the first face-driven data generated image, the method further includes: The face-driven image data samples and the first virtual face generated image are input into a pre-trained discriminator to obtain a first authenticity judgment result for the face-driven image data samples and a second authenticity judgment result for the first virtual face generated image; based on the first authenticity judgment result and the second authenticity judgment result, the model parameter information of the first sub-face processing model is adjusted until the second authenticity judgment result of the first virtual face generated image matches the first authenticity judgment result of the face-driven image data samples; and / or The virtual face image sample and the image generated from the first face-driven data are input into a pre-trained discriminator to obtain a third authenticity judgment result for the virtual face image sample and a fourth authenticity judgment result for the image generated from the first face-driven data. Based on the third authenticity judgment result and the fourth authenticity judgment result, the model parameter information of the first sub-face processing model is adjusted until the fourth authenticity judgment result of the image generated from the first face-driven data matches the third authenticity judgment result of the virtual face image sample.

6. The method according to claim 1, characterized in that, The second sub-face processing model is trained through the following steps: The virtual face image sample is input into the encoder of the first sub-face processing model that has been trained to obtain the third virtual expression feature; The third virtual expression feature is input into the second sub-face processing model to obtain the predicted BS coefficients corresponding to the virtual face image sample; Based on the predicted BS coefficients and the known BS ​​coefficients corresponding to the virtual face image samples, the fifth loss information is determined. Based on the fifth loss information, the model parameter information of the second sub-face processing model is adjusted.

7. The method according to claim 1, characterized in that, The acquisition of face-driven image data samples and virtual face image samples includes: Acquire raw face-driven image data and raw virtual face image; Augmentation processing is performed on the original face-driven image data and the original virtual face image respectively to obtain augmented face-driven image data samples and augmented virtual face image samples. The augmented face-driven image data and the augmented virtual face image are segmented to obtain face-driven image data samples containing the first face region and virtual face image samples containing the second face region.

8. The method according to claim 7, characterized in that, The step of segmenting the augmented face-driven image data and the augmented virtual face image to obtain face-driven image data samples containing a first face region and virtual face image samples containing a second face region includes: Face detection, face key point detection and face registration are performed sequentially on the augmented face-driven image data and the augmented virtual face image to determine the first face region of the augmented face-driven image data and the second face region of the augmented virtual face image. Based on the first face region, the augmented face-driven image data is segmented to obtain a face-driven image data sample containing the first face region. Based on the second face region, the augmented virtual face image is segmented to obtain a virtual face image sample containing the second face region.

9. A facial expression capture device, characterized in that, include: The first acquisition module is used to acquire face-driven image data; The first extraction module is used to perform a first feature extraction on the face-driven image data to obtain the first expression feature of the face-driven image data; The second extraction module is used to generate a virtual face image based on the first expression feature, and to extract a second feature from the virtual face image to obtain the second expression feature of the virtual face image. The generation module is used to generate shape fusion (BS) coefficients based on the second facial expression features; The BS coefficients are used as input to the 3D game engine to generate a 3D virtual face model. The BS coefficients are generated by inputting the face-driven image data into a pre-trained face processing model. The face processing model includes a first sub-face processing model and a second sub-face processing model; the first sub-face processing model is used to output the second expression feature based on the face-driven image data, and the second sub-face processing model is used to obtain the BS coefficient based on the second expression feature; The first sub-face processing model includes an encoder, a first decoder, and a second decoder. The encoder is used to extract features from the image to obtain expression features. The first decoder is used to decode the expression features to obtain a virtual face generation image. The second decoder is used to decode the expression features to obtain a face-driven data generation image. The device further includes: The second acquisition module is used to acquire face-driven image data samples and virtual face image samples; The first input module is configured to encode the face-driven image data sample through the encoder of the first sub-face processing model to obtain a first controller expression feature, input the first controller expression feature into the first decoder of the first sub-face processing model to obtain a first virtual face generated image; and to encode the virtual face image sample through the encoder of the first sub-face processing model to obtain a first virtual expression feature, input the first virtual expression feature into the second decoder of the first sub-face processing model to obtain a first face-driven data generated image; The encoding module is used to encode the first virtual face generated image through the encoder to obtain a second virtual expression feature; and to encode the first face driving data generated image through the encoder to obtain a second controller expression feature; The first adjustment module is used to adjust the model parameter information of the first sub-face processing model based on the first loss information between the first control personnel's facial expression features and the second virtual facial expression features, and the second loss information between the first virtual facial expression features and the second control personnel's facial expression features.

10. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the facial expression capture method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the facial expression capture method as described in any one of claims 1 to 8.

Images