Three-dimensional face generation method and apparatus, three-dimensional face reenactment method and apparatus

Abstract

The application discloses a three-dimensional face generation method and device, a three-dimensional face reenactment method and device, wherein the generation method comprises the following steps: acquiring an expression vector of a first image containing a first identity object based on a target expression coding network model; the first image is an image with a face having an occlusion; inputting the expression vector and a second image containing a second identity object into a target face generation network model to determine a target image of migrating the expression vector of the first image to a face of the second identity object in the second image; and enabling the face region of the first identity object in the first image to be identified as the expression vector even in the case of having the occlusion and to be migrated to the face of the second identity object in the second image, so that the accuracy of face expression migration is improved, and the robustness of face expression migration in the case of having the occlusion is improved.

Description

Technical Field

[0001] This application relates to the field of computer applications, specifically to a method and apparatus for generating and recreating three-dimensional faces. This application also relates to a computer storage medium and an electronic device. Background Technology

[0002] Face replay technology refers to transferring the facial expressions and head postures of one person (source face) onto another person's face (target face) while maintaining the identity of the target face. Face replay technology has a wide range of applications, such as creating virtual anchors and virtual idols. Simply put, given a video of a source face speaking and a static image of the target face, face replay technology can dynamically generate a new video of the target face from the static image, referencing the source face video. This new video will have the same facial expressions and head posture changes as the source video. Summary of the Invention

[0003] This application provides a three-dimensional face generation method to solve the problem in the prior art that face generation is impossible or inaccurate due to occlusion of the face, and further, the problem of not being able to realize three-dimensional face replay.

[0004] This application provides a three-dimensional face generation method, including:

[0005] Based on the target expression encoding network model, the expression vector of the first image containing the first identity object is obtained, wherein the target expression encoding network model is a model trained and determined based on the expression image of the face with occlusion; the first image is an image of the face with occlusion.

[0006] The expression vector and the second image containing the second identity object are input into the target face generation network model to determine the target image of the second identity object's face in the second image to transfer the expression vector of the first image. The target face generation network model is a model determined by expression transfer training based on different expressions corresponding to images with the same identity.

[0007] In some embodiments, the method further includes:

[0008] Obtain a target sample image, a positive sample image with an expression similar to that of the target sample image, and a negative sample image with an expression dissimilar to that of the target sample image from an expression image sample dataset. In this case, at least one of the target sample image, the positive sample image, and the negative sample image has a partial facial region that is occluded, and the target sample image, the positive sample image, and the negative sample image are sample images with different identity information.

[0009] The initial expression coding network model is trained based on the target sample image, the positive sample image, and the negative sample image. The expression coding model is used to calculate the expression vector corresponding to the input image.

[0010] When the preset model convergence condition is met based on the expression vector, the target expression encoding network model is obtained.

[0011] In some embodiments, training the initial facial expression encoding model based on the target sample image, the positive sample image, and the negative sample image includes:

[0012] The target sample image, the positive sample image, and the negative sample image are respectively input into the initial expression encoding network model to obtain the corresponding expression vectors;

[0013] Based on the expression vectors corresponding to the target sample image, the expression vectors corresponding to the positive sample image, the expression vectors corresponding to the negative sample image, and a preset loss function, a loss value is obtained;

[0014] Based on the loss value, determine whether the preset convergence condition is met. If not, adjust the parameters of the initial facial expression encoding network model and train the adjusted facial expression encoding network model for the next round.

[0015] In some embodiments, inputting the expression vector and a second image containing the second identity object into a target face generation network model to determine the target image for transferring the expression vector of the first image to the face of the second image includes:

[0016] The second image is input into the encoder of the target face generation network model to determine the facial features corresponding to the second image;

[0017] The expression vector of the first image and the facial features corresponding to the second image are input into the embedding module of the target face generation network model to determine that the expression vector of the first image is embedded into the target facial features of the second image.

[0018] The target facial features are input into the decoder of the target face generation network model to determine the target image.

[0019] In some embodiments, it also includes:

[0020] Based on the target pose encoding network model, the head pose vector of the first image is obtained, wherein the target pose encoding network model is a model determined by training on a head pose image with occlusions.

[0021] The step of inputting the expression vector and the second image containing the second identity object into the target face generation network model to determine the target image for transferring the expression vector of the first image to the second image further includes: inputting the head pose vector into the target face generation network model that has the expression vector and the second image as input, and determining the target image for transferring the expression vector and the head pose vector to the face of the second identity object in the second image.

[0022] In some embodiments, the method further includes:

[0023] Acquire a head pose image with occlusions, the head pose image having a pose ground truth label;

[0024] The head pose image is input into the initial pose encoding network model to determine the predicted pose label;

[0025] Based on the true pose label, the predicted pose label, and the preset loss function, the loss value is obtained;

[0026] Based on the loss value, determine whether the preset convergence condition is met. If not, adjust the parameters of the initial pose encoding network model and train the adjusted initial pose encoding model in the next round until the preset convergence condition is met, and obtain the target pose encoding network model.

[0027] In some embodiments, the method further includes:

[0028] The third image containing the third identity object is input into the target expression encoding network model to obtain the expression vector of the third image;

[0029] The fourth image containing the third identity object is input into the encoder of the initial face generation network model to determine the facial features corresponding to the fourth image.

[0030] The expression vector of the third image and the facial features corresponding to the fourth image are input into the embedding module of the initial face generation network model to determine the facial features embedded with the expression vector of the third image.

[0031] The facial features embedded with the expression vector of the third image are input into the decoder of the initial face generation network model to determine the fifth image embedded with the expression vector of the third image.

[0032] Determine whether the loss value between the fifth image and the third image satisfies the preset convergence condition;

[0033] If not, the initial face generation network model is trained again based on the reconstructed loss function until the convergence condition is met.

[0034] In some embodiments, the method further includes:

[0035] The third image containing the third identity object is input into the target head pose encoding network model to obtain the head pose vector of the third image;

[0036] The fourth image containing the third identity object is input into the encoder of the initial face generation network model to determine the facial features corresponding to the fourth image.

[0037] The head pose vector of the third image is input into the embedding module of the initial face generation network model to determine the facial features embedded with the head pose vector of the third image.

[0038] The step of inputting the facial features embedded with the expression vector of the third image into the decoder of the initial face generation network model to determine the fifth image embedded with the expression vector of the third image includes:

[0039] The facial features, including the expression vector and head pose vector embedded in the third image, are input into the decoder of the initial face generation network model to determine the fifth image, which is embedded with the expression vector and head pose vector.

[0040] This application also provides a three-dimensional face generation device, including:

[0041] The acquisition unit is used to acquire the expression vector of a first image containing a first identity object based on a target expression encoding network model, wherein the target expression encoding network model is a model trained and determined based on an expression image of a face with occlusions; the first image is an image of a face with occlusions.

[0042] The determining unit is used to input the expression vector and the second image containing the second identity object into the target face generation network model to determine the target image to transfer the expression vector of the first image to the face of the second image, wherein the target face generation network model is a model determined by expression transfer training based on different expressions corresponding to images with the same identity.

[0043] This application also provides a three-dimensional facial reconstruction method, including:

[0044] Acquire video information containing a first identity object and a static facial image containing a second identity object; wherein, the video information includes video frame images of faces with occlusions;

[0045] The extracted video frame images are input into the target facial expression coding network model to obtain the facial expression vectors of the video frame images;

[0046] Based on the playback sequence of the video information, the expression vector and the target static image are input into the face generation network model to obtain the target face sequence frame image that transfers the expression vector to the face of the target static image;

[0047] The target facial sequence frame images are merged according to the playback sequence to obtain a facial replay video on which the expression vectors are transferred to the target static image.

[0048] In some embodiments, it also includes:

[0049] The extracted video frame images from the video information are input into the target pose coding network model to obtain the head pose vector of the video frame images;

[0050] Based on the playback sequence of the video information, the expression vector and the target static image are input into a face generation network model to generate a target facial sequence frame image that transfers the expression vector to the face of the target static image, and the model further includes:

[0051] Based on the playback sequence of the video information, the expression vector, the head pose vector, and the target static image are input into the face generation network model to obtain a target facial sequence frame image that transfers the expression vector and the head pose vector to the face of the target static image.

[0052] This application also provides a three-dimensional facial reconstruction device, comprising:

[0053] The first acquisition unit is used to acquire video information containing a first identity object and a static facial image containing a second identity object; wherein, the video information includes video frame images of faces with occlusions;

[0054] The second acquisition unit is used to input the video frame images extracted from the video information into the target expression coding network model to obtain the expression vectors of the video frame images;

[0055] The third acquisition unit is used to input the expression vector and the target static image into the face generation network model according to the playback sequence of the video information, and acquire the target face sequence frame image to transfer the expression vector to the face of the target static image.

[0056] The replay unit is used to merge the target facial sequence frame images according to the playback sequence to determine the facial replay video on the target static image by transferring the expression vector.

[0057] This application also provides a computer storage medium for storing data generated by a network platform, and a program for processing the data generated by the network platform.

[0058] When the program is read and executed by the processor, it performs the three-dimensional face generation method as described above; or, it performs the three-dimensional face reenactment method as described above.

[0059] This application also provides an electronic device, including:

[0060] processor;

[0061] The memory is used to store programs that process data generated by the network platform. When the program is read and executed by the processor, it performs the three-dimensional face generation method described above; or, it performs the three-dimensional face replay method described above.

[0062] Compared with the prior art, this application has the following advantages:

[0063] In a 3D face generation method provided in this application, an expression vector containing a first image of a first identity object can be obtained through a target expression coding network model. The first image is an image of a face with occlusions. The expression vector of the occluded image can be obtained through the target expression coding network model, and the expression vector obtained by the target expression coding network model is not affected by identity information, i.e., identity decoupling is achieved, so that the obtained expression vector is not limited by identity, improving the accuracy and efficiency of expression recognition. By inputting the expression vector and a second image containing a second identity object into the target face generation network model, the expression vector of the first image is determined to be transferred to the target image of the second identity object's face in the second image. This ensures that the expression vector of the first identity object's face region in the first image can still be recognized even with occlusions and transferred to the second identity object's face in the second image, thereby improving the accuracy of facial expression transfer and enhancing the robustness of facial expression transfer under occlusion conditions.

[0064] This application also provides a three-dimensional facial replay method. This method inputs video frame images from the video information of a first identity object into a target expression coding network model to obtain expression vectors from the video frame images. The video frame images can be images of the first identity object's face with occlusions. Based on the expression vectors and a static facial image including a second identity object input into a facial generation network model, a target facial sequence frame image of the second identity object's face, to which the expression vectors are transferred to the static facial image, is obtained. This image is then merged according to the video playback sequence to obtain a facial replay video with the expression vectors transferred to the static facial image, ensuring the accuracy and robustness of facial expression transfer in the facial replay video. Attached Figure Description

[0065] Figure 1 This is a flowchart of an embodiment of a three-dimensional face generation method provided in this application;

[0066] Figure 2 This is a schematic diagram of the structure of the expression encoding network model training in an embodiment of a three-dimensional face generation method provided in this application;

[0067] Figure 3 This is a schematic diagram of the training structure of the pose encoding network model in an embodiment of a three-dimensional face generation method provided in this application;

[0068] Figure 4 This is a schematic diagram of the structure of the face generation network model training in an embodiment of a three-dimensional face generation method provided in this application;

[0069] Figure 5 This is a schematic diagram of the structure of an embodiment of a three-dimensional face generation device provided in this application;

[0070] Figure 6 This is a schematic diagram of the structure of an embodiment of a three-dimensional facial generation model provided in this application;

[0071] Figure 7 This is a flowchart of an embodiment of a three-dimensional face reconstruction method provided in this application;

[0072] Figure 8 This is a schematic diagram of the structure of an embodiment of a three-dimensional facial replay device provided in this application;

[0073] Figure 9 This is a schematic diagram of the structure of an embodiment of an electronic device provided in this application. Detailed Implementation

[0074] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0075] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The descriptive terms used in this application and the appended claims, such as "a," "first," and "second," are not intended to limit quantity or sequence, but rather to distinguish information of the same type from one another.

[0076] Based on the background technology described above, the purpose of face reconstruction is to transfer the expression and head posture of a source face to a target face. Existing technologies for face reconstruction can generally include two types: methods based on three-dimensional face reconstruction coefficients and methods based on facial key points.

[0077] In the method based on 3D face reconstruction coefficients, the facial expression and head pose are represented by the coefficients of the 3D face. The method first performs 3D face reconstruction on the source face that provides the expression and pose, then obtains the reconstructed expression coefficients and head pose coefficients, and then embeds the expression coefficients and head pose coefficients into the face generator to control the face generator to generate a target face with a specific expression and pose.

[0078] In the method based on facial landmarks, the facial landmarks of the source face are first obtained through the facial landmark module. The landmarks include facial expression information or facial expression information and head pose information. Then, the facial landmarks are embedded into the face generator in a corresponding manner to generate the target face.

[0079] When a face is occluded, both the three-dimensional face reconstruction coefficient method and the face key point method will lead to failure or inaccurate reconstruction of the three-dimensional face because the relevant information of the occluded part cannot be detected or is detected inaccurately.

[0080] To address the aforementioned issues, this application provides a three-dimensional face generation method, a three-dimensional face generation network model, and a model training method, in order to overcome the problems in existing technologies where inaccurate information collection due to facial occlusion leads to face generation failure or inaccuracy in subsequent replay. A detailed description follows.

[0081] like Figure 1 As shown, Figure 1 This is a flowchart illustrating an embodiment of a three-dimensional face generation method provided in this application; the embodiment includes:

[0082] Step S101: Based on the target expression encoding network model, obtain the expression vector of the first image containing the first identity object, wherein the target expression encoding network model is a model trained and determined based on the expression image of the face with occlusion; the first image is an image of the face with occlusion.

[0083] Step S102: Input the expression vector and the second image containing the second identity object into the target face generation network model to determine the target image of the second identity object's face in the second image to transfer the expression vector of the first image. The target face generation network model is a model determined by expression transfer training based on different expressions corresponding to images with the same identity.

[0084] Based on step S101 above, obtaining the expression vector requires a target expression encoding network model, which is a pre-trained expression encoding network model. The first image of the first identity object can be a frame image containing facial information from video information, i.e., a frame image of a face; it can also be an image containing facial information from a static or dynamic image. Specifically, the first image can be an image with facial occlusion (also called an occlusion area). In this embodiment, the first image can be an extracted video frame image from the video information, for example, an extracted JPEG image. It is understood that the video frame image with occlusion can be present throughout the entire video information, or it can be a partial video frame with facial occlusion.

[0085] The following is combined with Figure 2 This paper describes the training process of the expression encoding network model in a three-dimensional face generation method provided in this application.

[0086] like Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of the expression encoding network model training in an embodiment of the three-dimensional face generation method provided in this application. The specific implementation process may include:

[0087] Step S101-1: Obtain a target sample image, a positive sample image with an expression similar to that of the target sample image, and a negative sample image with an expression dissimilar to that of the target sample image from the facial expression image sample dataset. In this case, at least one of the target sample image, the positive sample image, and the negative sample image has a partial facial area that is obscured, and the target sample image, the positive sample image, and the negative sample image are sample images with different identity information.

[0088] The facial expression image sample dataset can be obtained from publicly available facial expression datasets or by searching for relevant facial expression image sample datasets using facial expression keywords. Facial expressions can be information displayed on the face that represents changes in facial emotions such as joy, anger, sorrow, and happiness.

[0089] Since facial expressions, which represent changes in facial emotions, share similarity, similarity can be assigned to facial expression images in a collected dataset of facial expression samples. The similarity of the facial expression images is labeled. For example, if the dataset includes facial expression images A, B, C, and D, then: image A and image B have relatively similar expressions, with a label value of 1; image C and image A have fairly similar expressions, with a label value of 2; image D and image A have generally similar expressions, with a label value of 0.5; and when the similarity between the images cannot be determined, the label value is 0. The facial expression images labeled with similarity are then identified as facial expression similarity image sample data.

[0090] The occlusion can be added randomly to the facial expression image sample data, and the form and number of additions are not limited.

[0091] Step S101-2: Train the initial expression coding network model based on the target sample image, the positive sample image and the negative sample image. The expression coding model is used to calculate the expression vector corresponding to the input image.

[0092] Step S101-3: When the preset model convergence condition is met based on the expression vector, the target expression encoding network model is obtained.

[0093] Step S102, which trains the initial expression coding model based on the target sample image, the positive sample image, and the negative sample image, may include:

[0094] Steps S101-21: Input the target sample image, the positive sample image, and the negative sample image into the initial expression encoding network model respectively to obtain the corresponding expression vectors;

[0095] Steps S101-22: Obtain the loss value based on the expression vector corresponding to the target sample image, the expression vector corresponding to the positive sample image, the expression vector corresponding to the negative sample image, and the preset loss function;

[0096] Step S101-23: Determine whether the preset convergence condition is met based on the loss value. If not, adjust the parameters of the initial facial expression encoding network model and train the adjusted facial expression encoding network model for the next round.

[0097] In this embodiment, loss data (or loss values) can be determined by comparing loss functions or ternary loss functions. The facial expression encoding network model is then optimized based on this loss data until it meets the convergence requirements. The convergent facial expression encoding network model is then identified as the target facial expression encoding network model. In this embodiment, a similarity-based ternary loss function is used, as follows:

[0098] L expression = maz(||f(a)-f(p)|| 2 -||f(a)-f(n)|| 2 +α,0)

[0099] Here, a, p, and n represent expression images a, p, and n, respectively, i.e., anchor images, positive images, and negative images. Positive and negative images represent more similar expressions, while negative images represent expressions with lower similarity or greater distance. The loss value of the ternary loss function indicates whether the predicted value of the expression encoding network model for learning expression image similarity is close to the labeled true similarity value. If it is close to or equal to the actual similarity label value, the expression encoding network model has converged; otherwise, it needs optimization. The optimization process involves adjusting the model parameters according to the ternary loss function until the output predicted value is close to or equal to the actual similarity label value, thus determining the adjusted expression encoding network model as the target expression encoding network model.

[0100] It should be noted that, in this embodiment, the input to the expression encoding network model can be an image, and the output is an expression vector. The mapping between expressions and vectors is achieved by constraining the distance between feature vectors.

[0101] Based on the above, the specific content of step S102 may include:

[0102] Step S102-11: Input the second image into the encoder of the target face generation network model to determine the facial features corresponding to the second image;

[0103] Step S102-12: Input the expression vector of the first image and the facial features corresponding to the second image into the embedding module of the target face generation network model to determine that the expression vector of the first image is embedded into the target facial features of the second image.

[0104] Step S102-13: Input the target facial features into the decoder of the target face generation network model to determine the target image.

[0105] To further improve the accuracy and robustness of the generated target image, the head pose of the first image can be used as a generation parameter for the target image. Specifically, this may further include:

[0106] Step S101-a1: Based on the target pose encoding network model, obtain the head pose vector of the first image, wherein the target pose encoding network model is a model determined by training on a head pose image with occlusions.

[0107] Step S102-12 may further include: inputting the head pose vector into the target face generation network model that has the expression vector and the second image input, and determining the target image of the second identity object's face that will transfer the expression vector and the head pose vector to the second image.

[0108] The target pose encoding network model can be a pre-trained model, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the training structure of the pose encoding network model in an embodiment of a three-dimensional face generation method provided in this application.

[0109] In this embodiment, training the target pose encoding network model may include:

[0110] Step Sa1: Obtain a head pose image with occlusion, the head pose image having a ground truth label; wherein, the head pose image can be obtained from publicly published head pose datasets, or by searching for relevant head pose keywords. Typically, the obtained head pose image can carry a ground truth label, such as: pitch, yaw, roll, i.e., the Euler angles of the head orientation.

[0111] Step Sa2: Input the head pose image into the initial pose encoding network model to determine the predicted pose label; wherein, the head pose image is input into the initial pose encoding network model, and the Euler angles of the head orientation are output, namely yaw, pitch, and roll. In this embodiment, these three Euler angles, combined with the head pose, can be understood as the angles of shaking the head left and right, nodding and raising the head, and tilting the head left and right, etc.

[0112] Step Sa3: Based on the ground truth pose label, the predicted pose label, and a preset loss function, obtain the loss value; wherein, the loss value is determined by using a Euclidean distance loss function. The Euclidean distance loss function can be:

[0113]

[0114] Among them, the parameters with the horizontal line are the predicted values ​​output by the attitude coding network model, and the parameters without the horizontal line are the true attitude labels, i.e., the true attitude label values.

[0115] Step Sa4: Determine whether the preset convergence condition is met based on the loss value. If not, adjust the parameters of the initial pose encoding network model and perform the next round of training on the adjusted initial pose encoding model until the preset convergence condition is met, thus obtaining the target pose encoding network model. According to the above Euclidean distance loss function, when the L... pose Meeting the model convergence requirement means that the closer the predicted values ​​output by the initial pose encoding network model are to the true pose labels, the better the target pose encoding network model is, and the better its robustness. Conversely, if the predicted values ​​are not close to the true pose labels, parameter adjustments are made and model training continues; this process can be iterative.

[0116] Based on the above steps S101-a1 and S101-a2, step S102 may further include:

[0117] Step S102-a1: Input the second image into the encoder of the target face generation network model to determine the facial features corresponding to the second image;

[0118] Step S102-a2: Input the expression vector, the head pose vector, and the facial features into the embedding module of the face generation network model to determine the target facial features of the second image to be embedded into the expression vector and head pose vector of the first image.

[0119] Step S102-a3: Input the target facial features into the decoder of the face generation network model to determine the target image.

[0120] Steps S102-a1 to S102-a3 can be understood as inputting expression vectors, head pose feature vectors, and facial features into a face generation network model to obtain a target image that embeds the expression vectors and head pose vectors of the first image into the second image. In other words, the target image is a second image containing the expression features and head pose features from the first image.

[0121] It is understood that the face generation network model can also be a pre-trained model, and the specific training process can be referred to as shown in Figure 4. Figure 4 As shown, it includes:

[0122] Step S10b1: Input the third image containing the third identity into the target expression encoding network model to obtain the expression vector of the third image;

[0123] Step S10b2: Input the fourth image containing the third identity into the encoder of the initial face generation network model to determine the facial features corresponding to the fourth image;

[0124] Step S10b3: Input the expression vector of the third image and the facial features corresponding to the fourth image into the embedding module of the initial face generation network model to determine the facial features embedded with the expression vector of the third image;

[0125] Step S10b4: Input the facial features embedded with the expression vector of the third image into the decoder of the initial face generation network model to determine the fifth image embedded with the expression vector of the third image;

[0126] Step S10b5: Determine whether the loss value between the fifth image and the third image meets the preset convergence condition;

[0127] Step S10b6: If not, then train the initial face generation network model in the next round according to the reconstructed loss function until the convergence condition is met.

[0128] Accordingly, in addition to obtaining the expression vector, the head pose vector can also be obtained to further improve the robustness of the face generation network model. Therefore, it can further include:

[0129] Step S10c1: Input the third image containing the third identity into the target head pose encoding network model to obtain the head pose vector of the third image;

[0130] Step S10c2: Input the fourth image containing the third identity into the encoder of the initial face generation network model to determine the facial features corresponding to the fourth image;

[0131] Step S10c3: Input the head pose vector of the third image into the embedding module of the initial face generation network model to determine the facial features embedded with the head pose vector of the third image.

[0132] Based on the above steps S10c1 to S10c3, step S10b4 includes:

[0133] The facial features, including the expression vector and head pose vector embedded in the third image, are input into the decoder of the initial face generation network model to determine the fifth image, which is embedded with the expression vector and head pose vector.

[0134] It should be noted that the sample data used to train the initial face generation network model can be images of the same identity object or images of different identity objects. In this embodiment, to improve the accuracy of the output data of the face generation network model, images of the same identity object are used to train the model. That is, the images input into the target expression encoding network model and the target head pose encoding network model can be images of the same identity object, including the face. The images input into the initial face generation network model can be images of the same identity object as those input into the target expression encoding network model and the head pose encoding network model, both having the same identity but different expressions and head poses. Using images of different expressions and head poses of the same identity object to train the initial face generation network model can improve the accuracy of the output results. Of course, the initial face generation model is not limited to training with images of the same identity object. In steps 10b2 and S10c1, the fourth image of the third identity object can be input into the initial face generation network model. The initial face generation network model encodes the fourth image into hidden layer feature data, and the hidden layer feature data is determined as the facial feature data corresponding to the fourth image. In this embodiment, the fourth image can be a static image that includes facial information, typically a static image that includes complete facial information.

[0135] In this embodiment, the facial feature data embedded with the expression vector, or the expression vector and the head pose vector, is decoded to map the facial feature data into an image for output. In this embodiment, by employing a pixel reconstruction loss function and / or a perceptual loss function, it is determined whether the output image result satisfies the model convergence requirement of the initial face generation network model, i.e., whether the output image matches the third image of the third-party identity object.

[0136] The pixel reconstruction loss function can be:

[0137]

[0138] The perception loss function can be:

[0139]

[0140] In the above formula, the The output image of the model is y, and the ground truth image is y. F represents the pre-trained perceptual network such as VGGNet. The pixel reconstruction loss function and the perceptual loss function can be selected separately. To further ensure the clarity of the output image, the perceptual loss function can be combined with the pixel reconstruction loss function to determine whether the face generation network model meets the convergence requirements. The convergence requirements can be based on whether the loss function decreases sufficiently and tends to stabilize; or whether the image generated by the face generation network model matches the source face image. If so, the convergence requirements are met; if not, the model parameters of the face generation network model are adjusted according to the loss function to optimize the model so that it meets the convergence requirements. Once the convergence requirements are met, it is determined as the target face generation network model. In this embodiment, the reconstruction of the loss function can be determined based on the input image, and is not limited to the pixel reconstruction loss function and the perceptual loss function mentioned above. For example, when the input image is different expressions of the same person in the same environment, or images of expressions and postures, the pixel reconstruction loss function can be omitted, and other parameters in the model can be adjusted.

[0141] The above is a description of an embodiment of a three-dimensional face generation method provided in this application. Through the target expression encoding network model and the target pose encoding network model in this embodiment, the expression feature data (expression vector) and head pose feature data (head pose vector) in the first image of the first identity object with occlusion on the face can be determined, and the expression feature data and head pose feature data are embedded into the target face feature data in the target face generation network model. Based on the target face generation network model, the target image that transfers the expression feature data, or the expression feature data and head pose feature data, to the second image of the second identity object can be output. The accuracy of the embedded data will not be affected by the presence of occlusion in the first image of the first identity object, thus preventing the generated target image from being unclear or inaccurate.

[0142] The above is a detailed description of an embodiment of a three-dimensional face generation method provided in this application. Corresponding to the aforementioned embodiment of a three-dimensional face generation method, this application also discloses an embodiment of a three-dimensional face generation device. Please refer to [link / reference]. Figure 5 Since the device embodiments are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments. The device embodiments described below are merely illustrative.

[0143] like Figure 5 As shown, Figure 5 This is a schematic diagram of an embodiment of a three-dimensional face generation device provided in this application; the training device embodiment includes:

[0144] The acquisition unit 501 is used to acquire the expression vector of a first image containing a first identity object based on a target expression encoding network model, wherein the target expression encoding network model is a model trained and determined based on an expression image of a face with occlusions; the first image is an image of a face with occlusions.

[0145] The determining unit 502 is used to input the expression vector and the second image containing the second identity object into the target face generation network model to determine the target image to transfer the expression vector of the first image to the face of the second image, wherein the target face generation network model is a model determined by expression transfer training based on different expressions corresponding to images with the same identity.

[0146] Since the target expression encoding network model is a model trained on facial expression images with occlusions, this embodiment may include training an initial expression encoding network model to determine the target encoding network model, specifically including:

[0147] The sample acquisition unit is used to acquire a target sample image, a positive sample image with an expression similar to that of the target sample image, and a negative sample image with an expression dissimilar to that of the target sample image from an expression image sample dataset. In this case, at least one of the target sample image, the positive sample image, and the negative sample image has a partial facial region with an occlusion, and the target sample image, the positive sample image, and the negative sample image are sample images with different identity information.

[0148] An expression training unit is used to train an initial expression encoding network model based on the target sample image, the positive sample image, and the negative sample image. The expression encoding model is used to calculate the expression vector corresponding to the input image.

[0149] The model acquisition unit is used to obtain the target expression encoding network model when the preset model convergence condition is met based on the expression vector.

[0150] The training unit may include: an expression vector acquisition subunit, a loss value acquisition subunit, and a determination subunit;

[0151] The expression vector acquisition subunit is used to input the target sample image, the positive sample image, and the negative sample image into the initial expression encoding network model respectively, and obtain the corresponding expression vectors respectively.

[0152] The loss value acquisition subunit is used to obtain a loss value based on the expression vector corresponding to the target sample image, the expression vector corresponding to the positive sample image, the expression vector corresponding to the negative sample image, and a preset loss function.

[0153] The determining subunit is used to determine whether the preset convergence condition is met based on the loss value. If not, the parameters of the initial expression coding network model are adjusted, and the adjusted expression coding network model is trained in the next round.

[0154] In this embodiment, the determining unit 502 includes: a facial feature determining subunit, an input subunit, and a target image determining subunit;

[0155] The facial feature determination subunit is used to input the second image into the encoder of the target face generation network model to determine the facial features corresponding to the second image;

[0156] The input subunit is used to input the expression vector of the first image and the facial features corresponding to the second image into the embedding module of the target face generation network model, and determine to embed the expression vector of the first image into the target facial features of the second image.

[0157] The target image determination subunit is used to input the target facial features into the decoder of the target face generation network model to determine the target image.

[0158] To further improve the robustness of the face generation network model, this embodiment may also include: a pose vector acquisition unit and a pose input unit;

[0159] The pose vector acquisition unit is used to acquire the head pose vector of the first image based on the target pose encoding network model, wherein the target pose encoding network model is a model trained and determined based on a head pose image with occlusions.

[0160] The pose input unit is used to input the head pose vector into the target face generation network model to determine the target image of the second identity object's face that will be transferred from the expression vector and the head pose vector of the first image to the second image.

[0161] Since the target pose encoding network model is a model trained and determined based on facial expression images with occlusions, this embodiment may include training an initial pose encoding network model to determine the target pose encoding network model. Specifically, it may include: a pose image acquisition unit, a prediction label determination unit, a loss value acquisition unit, and a target pose determination unit.

[0162] The posture image acquisition unit is used to acquire a head posture image with occlusions, and the head posture image has a posture ground truth label.

[0163] The predicted label determination unit is used to input the head pose image into the initial pose coding network model to determine the predicted pose label;

[0164] The loss value acquisition unit is used to obtain a loss value based on the true pose label, the predicted pose label, and a preset loss function;

[0165] The target pose determination unit is used to determine whether a preset convergence condition is met based on the loss value. If not, the parameters of the initial pose encoding network model are adjusted, and the adjusted initial pose encoding model is trained in the next round until the preset convergence condition is met, thereby obtaining the target pose encoding network model.

[0166] Based on the determination unit 502, it can be known that the target face generation network model is a model determined by expression transfer training based on different expressions corresponding to images with the same identity. Therefore, this embodiment may also include: an expression vector acquisition unit, a facial feature determination unit, an input unit, a fifth image determination unit, a loss determination unit, and a function reconstruction unit.

[0167] The expression vector acquisition unit is used to input a third image containing a third identity object into the target expression encoding network model to obtain the expression vector of the third image;

[0168] The facial feature determination unit is used to input the fourth image containing the third identity object into the encoder in the initial facial generation network model to determine the facial features corresponding to the fourth image.

[0169] The input unit is used to input the expression vector of the third image and the facial features corresponding to the fourth image into the embedding module of the initial face generation network model to determine the facial features embedded with the expression vector of the third image.

[0170] The fifth image determination unit is used to input the facial features of the expression vector embedded in the third image into the decoder of the initial face generation network model to determine the fifth image embedded in the expression vector of the third image.

[0171] The loss determination unit is used to determine whether the loss value between the fifth image and the third image meets the preset convergence condition.

[0172] The function reconstruction unit is used to train the initial face generation network model in the next round according to the reconstructed loss function when the determination result of the loss determination unit is negative, until the convergence condition is met.

[0173] The above is a description of an embodiment of a three-dimensional face generation device provided in this application. For details of the above generation device embodiment, please refer to the detailed description of steps S101 to S102 in the above generation method embodiment. The corresponding or the same content will not be repeated here.

[0174] Based on the above, this application also provides a three-dimensional facial generation model, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of a three-dimensional face generation model embodiment provided in this application; specifically, it may include: an expression encoding network model 601 and a face generation network model 603; or it may include: an expression encoding network model 601, a pose encoding network model 602 and a face generation network model 603; the following description uses the expression encoding network model 601, the pose encoding network model 602 and the face generation network model 603 as examples.

[0175] The expression encoding network model 601 is used to output facial expression feature data (also called expression vector or expression feature vector) of the first identity object based on the first image of the first identity object input; wherein, the first image includes an image of the face with occlusions. The training process of the expression encoding network model 601 has been described in the above generation method and will not be detailed here.

[0176] The pose encoding network model 602 is used to output head pose feature data (also referred to as head pose vector or head pose feature vector) of the head of the first identity object based on the first image of the first identity object as input. The training process of the pose encoding network model 602 has been described in the above generation method and will not be detailed here.

[0177] The face generation network model 603 is used to transfer the input facial expression feature data and head pose feature data, as well as the second image of the second identity object, to the target image of the second identity object's face.

[0178] The face generation network model 603 may include: encoder 603-1, embedding module 603-2, and decoder 603-3.

[0179] The encoder 603-1 is used to encode the second image of the input second identity object and output the facial feature data corresponding to the second image.

[0180] The embedding module 603-2 is used to embed the acquired facial expression feature data, or facial expression feature data and head posture feature data, into the facial feature data;

[0181] The decoder 603-3 is used to decode the feature data output by the embedding module 603-2 to generate a target image that transfers the facial expression feature data in the first image to the second image, or to generate a target image that transfers the facial expression feature data and the head posture feature data in the first image to the second image.

[0182] For details regarding the face generation network model 603, please refer to the description in the above generation method embodiments, which will not be elaborated here.

[0183] The above describes an embodiment of a 3D facial generation model provided in this application. This model can obtain facial expression feature data and head pose feature data from a first image containing a first identity object through an expression encoding network model and a pose encoding network model. The expression feature data is obtained based on the presence of occlusions on the face and head (or only the face) of the first image. Because the expression encoding network model is trained using images containing different identity objects, the model's output expression and head pose information is unaffected by facial and head occlusions, resulting in more accurate facial expression feature data and improved robustness of the expression encoding network model. Furthermore, embedding these two data points into the facial feature data of the second image of the second identity object obtained through encoder 603-1 achieves identity decoupling and ensures the accuracy of the generated 3D facial image.

[0184] Based on the above, this application also provides a three-dimensional facial reconstruction method, such as... Figure 7 As shown, Figure 7 This is a flowchart of an embodiment of a three-dimensional face reconstruction method provided in this application; the embodiment of the method may include:

[0185] Step S701: Obtain video information containing the first identity object and a static facial image containing the second identity object; wherein, the video information includes video frame images of faces with occlusions.

[0186] Step S702: Input the extracted video frame images from the video information into the target expression coding network model to obtain the expression vectors of the video frame images;

[0187] Step S703: Based on the playback sequence of the video information, input the expression vector and the target static image into the face generation network model to obtain the target facial sequence frame image to which the expression vector is transferred to the face of the target static image;

[0188] Step S704: Merge the target facial sequence frame images according to the playback sequence to obtain a facial replay video on which the expression vectors are transferred to the target static image.

[0189] In this embodiment, the relationship between the video information and the facial static image in step S701 can be understood as follows: the video information includes video information of the first identity speaking, and the facial expression information of the first identity object in the video information, or the facial expression information and head posture information, are transferred to the face of the second identity object in the facial static image.

[0190] In this embodiment, the video frame image with the obstruction can be a portion of the video frame image in the video information or all of the video frame images.

[0191] In this embodiment, the target expression encoding network model in step S702 is a pre-trained model, i.e., a model that meets the convergence and / or testing requirements. For details, please refer to the relevant content regarding the training process of the target expression encoding network model in the above-mentioned 3D face generation method; it will not be elaborated here. Expression vectors can be obtained by inputting video frame images from the video information into the target expression encoding network model, and may also include:

[0192] The extracted video frame images are input into the target pose coding network model to obtain the head pose vector of the video frame images.

[0193] In this embodiment, the target pose encoding network model can also be a pre-trained model, i.e., a model that meets the convergence and / or testing requirements. For details, please refer to the relevant content regarding the training process of the target pose encoding network model in the above-mentioned 3D face generation method; it will not be elaborated here.

[0194] Step S703

[0195] The specific implementation process in this embodiment can be as follows: the expression vector and the head pose vector are embedded into the hidden layer features (facial feature data of the video frame image), and then decoded into a sequence of frame images with expression vectors and head pose vectors. Specifically, the target static image is input into the encoder of the face generation network model to obtain facial feature data. Then, the expression vector and head pose vector are extracted from the embedding module of the face generation network model. The expression vector and head pose vector are then embedded into the facial feature data. Afterward, the decoder of the face generation network model decodes the embedded data to obtain a sequence of frame images with the expression vector and head pose vector transferred to the facial feature data.

[0196] Understandably, one can embed only the expression vector into the facial feature data, or one can embed both the expression vector and the head pose vector into the facial feature data.

[0197] Steps S701 to S704 described above can be combined with the content of steps S101 to S102 described above, as well as the description of the above-described 3D face generation network model embodiment, and will not be repeated here. The main purpose of this embodiment is that, by acquiring a video containing a first identity object and a static facial image containing a second identity object, the target expression encoding network model and the target pose encoding network model will extract expression feature data and head pose feature data from each video frame image in the order of video playback and input them into the target face generation network model to generate a sequence of frame images corresponding to the video playback sequence. The obtained sequence of frame images is then processed and merged into the final target video according to the playback sequence. The target video is the face replay video formed by transferring expression vectors and head pose vectors to the face of the second identity object in the static facial image. Because the expression encoding network model is a model trained on facial occlusion areas and is not restricted by identity, even if there are video frame images in the video with facial occlusions, the corresponding expression vectors and head pose vectors can still be obtained. Therefore, the accuracy of the face replay video or image can be improved.

[0198] It should be noted that, in this embodiment, the video frame images and facial still images in the video can be facial images of different people, that is, the first identity object is person A, and the video frame images can include person A's facial expressions and / or head posture; the second identity object is person B; or, they can be the same person, that is, the video frame images can be images of person A's facial expressions and / or head posture, and the facial still images can be images of person A including facial information. The above is only an illustrative description and is not intended to limit the video frame images and facial still images.

[0199] The above is a description of an embodiment of a three-dimensional face replay method provided in this application. The description is relatively brief, and the specific content can be combined with the description of the above-mentioned three-dimensional face generation method and device embodiments, as well as the description of the three-dimensional face generation network model embodiments.

[0200] Based on the above, this application also provides a three-dimensional facial reconstruction device, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of an embodiment of a three-dimensional facial reconstruction device provided in this application. The embodiment of the device includes:

[0201] The first acquisition unit 801 is used to acquire video information containing a first identity object and a static facial image containing a second identity object; wherein, the video information includes video frame images of faces with occlusions;

[0202] The second acquisition unit 802 is used to input the video frame images extracted from the video information into the target expression coding network model to obtain the expression vectors of the video frame images;

[0203] The third acquisition unit 803 is used to input the expression vector and the target static image into the face generation network model according to the playback sequence of the video information, and acquire the target face sequence frame image to transfer the expression vector to the face of the target static image.

[0204] The replay unit 804 is used to merge the target facial sequence frame images according to the playback sequence to determine the facial replay video on the target static image by transferring the expression vector.

[0205] For details regarding the above-described embodiments of the three-dimensional face replay device, please refer to the descriptions of the above-described embodiments of the three-dimensional face replay method, the three-dimensional face generation method and device, and the three-dimensional face generation network model, which will not be repeated here.

[0206] Based on the above, this application also provides a computer storage medium for storing data generated by a network platform and a program for processing the data generated by the network platform.

[0207] When the program is read and executed by the processor, it performs the steps as described in the above-described three-dimensional face generation method embodiment; or, it performs the steps as described in the above-described three-dimensional face reenactment method embodiment.

[0208] Based on the above, this application also provides an electronic device, such as... Figure 9 As shown, Figure 9 This is a schematic diagram of an embodiment of an electronic device provided in this application, which includes:

[0209] Processor 901;

[0210] The memory 902 is used to store a program for processing data generated by the network platform. When the program is read and executed by the processor, it performs the steps involved in the above-described three-dimensional face generation method embodiment; or, it performs the steps involved in the above-described three-dimensional face reenactment method embodiment.

[0211] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A three-dimensional face generation method characterized by comprising: The method comprises: obtaining video information containing a first identity object and a face static image containing a second identity object, wherein the video information comprises a video frame image with a face having an occlusion, and the video frame image is a first image; obtaining an expression vector of the first image containing the first identity object based on a target expression coding network model, wherein the target expression coding network model is a model determined based on training of expression images with faces having occlusions; inputting the expression vector and the face static image containing the second identity object as a second image into a target face generation network model according to a playing time sequence of the video information, to obtain a target face sequence frame image of the expression vector of the first image migrated to the second image, wherein the target face generation network model is a model determined based on expression migration training of images with the same identity corresponding to different expressions; merging the target face sequence frame image according to the playing time sequence to obtain a face replay video of the expression vector migrated to the face static image.

2. The method of claim 1, wherein, The method further comprises: obtaining a target sample image, a positive sample image similar in expression to the target sample image, and a negative sample image not similar in expression to the target sample image from an expression image sample data set, wherein at least one of the target sample image, the positive sample image, and the negative sample image has a partial face region with an occlusion, and the target sample image, the positive sample image, and the negative sample image are sample images with different identity information; training an initial expression coding network model based on the target sample image, the positive sample image, and the negative sample image, wherein the expression coding network model is used to calculate an expression vector corresponding to an input image; obtaining the target expression coding network model when it is determined that the expression vector satisfies a preset model convergence condition.

3. The method of claim 2, wherein, The training of the initial expression coding network model based on the target sample image, the positive sample image, and the negative sample image comprises: inputting the target sample image, the positive sample image, and the negative sample image into the initial expression coding network model respectively to obtain corresponding expression vectors respectively; obtaining a loss value based on the expression vector corresponding to the target sample image, the expression vector corresponding to the positive sample image, the expression vector corresponding to the negative sample image, and a preset loss function; determining whether the preset convergence condition is satisfied according to the loss value, and if not, adjusting parameters of the initial expression coding network model and performing next-round training on the adjusted expression coding network model.

4. The method of claim 1, wherein, The inputting of the expression vector and the face static image containing the second identity object as a second image into the target face generation network model according to the playing time sequence of the video information to obtain a target image of the expression vector of the first image migrated to the face of the second image comprises: inputting the second image into an encoder of the target face generation network model to determine a face feature corresponding to the second image; inputting the expression vector of the first image and the face feature corresponding to the second image into an embedding module of the target face generation network model, to determine embedding of the expression vector of the first image into the target face feature of the second image; inputting the target face feature into a decoder of the target face generation network model, to determine the target image.

5. The method of claim 1, wherein, Further comprising: obtaining a head posture vector of the first image based on a target posture encoding network model, wherein the target posture encoding network model is a model determined based on training of a head posture image with an occlusion; the inputting of the expression vector and the second image containing a second identity object into the target face generation network model to determine migration of the expression vector of the first image to the target image of the second image further comprises: inputting the head posture vector into the target face generation network model inputted with the expression vector and the second image, to determine migration of the expression vector and the head posture vector to the target image of the face of the second identity object of the second image.

6. The method of claim 5, wherein, The method further comprises: obtaining a head posture image with an occlusion, the head posture image having a posture ground truth label; inputting the head posture image into an initial posture encoding network model to determine a predicted posture label; obtaining a loss value based on the posture ground truth label, the predicted posture label and a preset loss function; determining whether the preset convergence condition is met according to the loss value, and if not, adjusting parameters of the initial posture encoding network model and performing next round of training on the adjusted initial posture encoding network model until the preset convergence condition is met, to obtain the target posture encoding network model.

7. The method of claim 1, wherein, The method further comprises: inputting a third image containing a third identity object into the target expression encoding network model to obtain an expression vector of the third image; inputting a fourth image containing a third identity object into an encoder of an initial face generation network model to determine a face feature corresponding to the fourth image; inputting the expression vector of the third image and the face feature corresponding to the fourth image into an embedding module of the initial face generation network model to determine the face feature embedded with the expression vector of the third image; inputting the face feature embedded with the expression vector of the third image into a decoder of the initial face generation network model to determine a fifth image embedded with the expression vector of the third image; determining whether a loss value between the fifth image and the third image meets a preset convergence condition; if not, performing next round of training on the initial face generation network model according to a reconstruction loss function until the convergence condition is met.

8. The method of claim 7, wherein, The method further comprises: inputting the third image containing the third identity object into a target posture encoding network model to obtain a head posture vector of the third image; inputting a fourth image containing a third identity object into an encoder of an initial face generation network model to determine a face feature corresponding to the fourth image; The head posture vector of the third image is input into an embedding module of the initial face generation network model to determine a face feature embedded with the head posture vector of the third image. The face feature embedded with the expression vector of the third image is input into a decoder of the initial face generation network model to determine a fifth image embedded with the expression vector of the third image, including: The face feature embedded with the expression vector and the head posture vector of the third image is input into a decoder of the initial face generation network model to determine a fifth image embedded with the expression vector and the head posture vector.

9. A three-dimensional face generating apparatus characterized by comprising: Including: An acquisition unit is configured to acquire video information containing a first identity object and a face static image containing a second identity object, the video information including a video frame image with a face having an occlusion, the video frame image being a first image, and acquire an expression vector of the first image containing the first identity object based on a target expression coding network model, wherein the target expression coding network model is a model determined based on training of an expression image with a face having an occlusion; A determination unit is configured to input the expression vector and the face static image containing the second identity object as a second image into a target face generation network model according to a playing time sequence of the video information, acquire a target face sequence frame image that migrates the expression vector of the first image to the second image, wherein the target face generation network model is a model determined based on expression migration training of different expressions corresponding to images with the same identity, and combine the target face sequence frame image according to the playing time sequence to acquire a face replay video that migrates the expression vector to the face static image. 10.A computer storage medium, configured to store network platform generated data and a program for processing the network platform generated data. The program, when read and executed by a processor, performs the three-dimensional face generation method in any one of claims 1-8. 11.An electronic device, comprising: a processor; a memory configured to store a program for processing network platform generated data, the program, when read and executed by the processor, performing the three-dimensional face generation method in any one of claims 1-8.

Images