Network training method and device for face reconstruction, equipment and storage medium

Abstract

Embodiments of the present application disclose a network training method and device for face reconstruction, equipment and a storage medium, which comprises: obtaining a plurality of first face images of a first target object; inputting each first face image into a corresponding coefficient regression network to obtain a face reconstruction coefficient of each first face image, each coefficient regression network having the same network parameters, the face reconstruction coefficient comprising an identity coefficient and an expression coefficient; obtaining a two-dimensional reconstruction image of the corresponding first face image according to the face reconstruction coefficient; constructing a loss function of the coefficient regression network according to the first face image, the corresponding two-dimensional reconstruction image and the face reconstruction coefficient, the loss function comprising an unsupervised training loss function and an identity constraint loss function of each first face image; and updating the network parameters of the coefficient regression network according to the loss function. The above method can solve the technical problem of low expression coefficient reconstruction accuracy when reconstructing a three-dimensional face image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a network training method, apparatus, device, and storage medium for face reconstruction. Background Technology

[0002] With the development of network communication technology (especially 5G), users can enjoy video calls, online classes, online meetings, and other network communication resources without leaving home using communication devices (such as mobile phones and computers). Currently, video communication is a common scenario in network communication technology, where both parties can see each other's current facial images. However, with the widespread use of video communication, users' security awareness is also increasing. In some scenarios, users do not want to show their real faces to others. In such cases, virtual cartoon images can be used instead of real faces.

[0003] When using cartoon images, the facial expressions and head movements of a user's current real-life portrait can be transferred to the cartoon image, making the cartoon image have the same facial expressions and head movements as the real-life portrait. In some related technologies, during the transfer process, a 3D face image is first reconstructed from the real-life portrait to obtain expression and pose coefficients, which are then transferred to the corresponding cartoon image. However, when reconstructing 3D face images, the accuracy of the 3D face image is usually the only focus, neglecting the accuracy of the expression coefficients. For example, a real-life portrait might have a squinting expression, but when reconstructing a 3D face image, the eyes might be directly identified as small eyes, failing to recognize the squinting expression. Thus, when the expression and pose coefficients of the 3D face image are transferred to the cartoon image, the cartoon image cannot mimic the squinting action, thereby reducing the accuracy of the cartoon image. Summary of the Invention

[0004] This application provides a network training method, apparatus, device, and storage medium for face reconstruction, in order to solve the technical problem of low accuracy of expression coefficient reconstruction when reconstructing three-dimensional face images in related technologies.

[0005] In a first aspect, one embodiment of this application provides a network training method for face reconstruction, comprising:

[0006] Acquire multiple frames of the first face image of the first target object;

[0007] Each frame of the first face image is input into the corresponding coefficient regression network to obtain the face reconstruction coefficients of each frame of the first face image. Each coefficient regression network has the same network parameters. The face reconstruction coefficients include identity coefficients and expression coefficients.

[0008] The two-dimensional reconstructed image of the first face image in the corresponding frame is obtained based on the face reconstruction coefficients;

[0009] The loss function of the coefficient regression network is constructed based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients. The loss function includes the unsupervised training loss function and the identity constraint loss function for each frame of the first face image.

[0010] The network parameters of the coefficient regression network are updated according to the loss function.

[0011] Secondly, one embodiment of this application also provides a network training device for face reconstruction, comprising:

[0012] The image acquisition module is used to acquire multiple frames of the first face image of the first target object;

[0013] An image input module is used to input each frame of the first face image into a corresponding coefficient regression network to obtain the face reconstruction coefficients of each frame of the first face image. Each coefficient regression network has the same network parameters, and the face reconstruction coefficients include identity coefficients and expression coefficients.

[0014] The image reconstruction module is used to obtain a two-dimensional reconstructed image of the first face image in the corresponding frame based on the face reconstruction coefficients.

[0015] The loss construction module is used to construct the loss function of the coefficient regression network based on the first face image, the corresponding two-dimensional reconstructed image and the face reconstruction coefficients. The loss function includes the unsupervised training loss function and the identity constraint loss function for each frame of the first face image.

[0016] The parameter update module is used to update the network parameters of the coefficient regression network according to the loss function.

[0017] Thirdly, one embodiment of this application also provides a network training device for face reconstruction, comprising:

[0018] One or more processors;

[0019] Memory, used to store one or more programs;

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the network training method for face reconstruction as described in the first aspect.

[0021] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the network training method for face reconstruction as described in the first aspect.

[0022] In one embodiment of this application, a technique is employed to obtain face reconstruction coefficients by acquiring multiple frames of a first face image containing the same target object and processing each frame of the first face image using coefficient regression networks with the same network parameters. Then, a two-dimensional reconstructed image corresponding to the first face image is reconstructed based on the face reconstruction coefficients. An unsupervised training loss function and an identity constraint loss function are constructed based on the first face image, the two-dimensional reconstructed image, and the face reconstruction coefficients. The network parameters of the coefficient regression network are then updated based on these two loss functions. This technique solves the technical problem of low accuracy in reconstructing expression coefficients when reconstructing three-dimensional face images. By having the coefficient regression network output identity coefficients and expression coefficients separately, the identity coefficients and expression coefficients can be determined independently. Furthermore, the identity constraint loss function ensures the accuracy of the identity coefficients, meaning that the coefficient regression network can obtain the same face identity based on different images of the same person, regardless of the expression, thus guaranteeing the accuracy of the expression coefficients. Meanwhile, in addition to constraining the accuracy of the identity coefficients, an unsupervised training loss function was constructed by combining the first face image, the 2D reconstructed image, and the face reconstruction coefficients. This improved the rationality of the loss function and, in turn, enhanced the performance of the coefficient sharing network when training it based on the loss function, further ensuring the accuracy of the expression coefficients. Attached Figure Description

[0023] Figure 1 A flowchart illustrating a network training method for face reconstruction provided in one embodiment of this application;

[0024] Figure 2 A flowchart illustrating a network training method for face reconstruction provided in one embodiment of this application;

[0025] Figure 3 A flowchart of a network training method for face reconstruction provided in one embodiment of this application;

[0026] Figure 4 A schematic diagram of a mutually exclusive emoji pair provided in one embodiment of this application;

[0027] Figure 5 A schematic diagram of the framework of a network training method for face reconstruction provided in one embodiment of this application;

[0028] Figure 6 A schematic diagram illustrating the application process of a coefficient regression network provided in one embodiment of this application;

[0029] Figure 7 This is a diagram of a virtual image generation framework provided in one embodiment of this application;

[0030] Figure 8A schematic diagram of the structure of a network training device for face reconstruction provided in one embodiment of this application;

[0031] Figure 9 This is a schematic diagram of the structure of a network training device for face reconstruction provided in an embodiment of this application. Detailed Implementation

[0032] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the drawings, not the entire structure.

[0033] When reconstructing 3D face images from real faces, both traditional methods and deep learning-based methods can be used. The traditional method projects facial landmarks from the real face and those in the reconstructed 3D face image onto the same 2D plane. It then fits a model based on the positional errors between semantically identical facial landmarks (e.g., the facial landmark representing the tip of the nose in the real face and the one representing the tip of the nose in the 3D face image have the same semantics). In other words, it adjusts the landmarks in the 3D face image to ensure that the 3D face image closely resembles the real face. The deep learning-based method trains a 3D face reconstruction model using a large number of 2D real face images, and then reconstructs the 3D face image using this model. Regardless of the method used to reconstruct the 3D face image, the accuracy of the facial expression coefficients cannot be guaranteed during reconstruction.

[0034] Based on this, this application provides a network training method, apparatus, device, and storage medium for face reconstruction, so as to improve the accuracy of expression coefficients when reconstructing three-dimensional face images.

[0035] One embodiment of this application provides a network training method for face reconstruction that can be executed by a network training device for face reconstruction. This network training device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. For example, the network training device for face reconstruction can be a computer, mobile phone, tablet computer, or interactive smart tablet, or other smart device.

[0036] Figure 1 This is a flowchart illustrating a network training method for face reconstruction, provided as an embodiment of this application. (Reference) Figure 1 The network training method for face reconstruction specifically includes:

[0037] Step 110: Obtain multiple frames of the first face image of the first target object.

[0038] The target object refers to the object whose face image needs to be reconstructed. Currently, the target object used in the neural network training process is denoted as the first target object. The neural network needs some coefficients to reconstruct the face based on the face image of the first target object. It can be understood that one or more first target objects can be used in a single training process; currently, we will take using one first target object as an example. Furthermore, different first target objects (i.e., objects with different faces) can be used in each training session. The face image of the first target object is denoted as the first face image. During a single training process, multiple frames of the first face image of the first target object are used; the number of frames can be set according to the actual situation. The method of obtaining the first face image is currently not limited. For example, multiple frames containing the face image of the first target object can be randomly selected from video data containing the first target object. Alternatively, the face region can be extracted from the selected face images containing the first target object, and the background region can be removed to obtain multiple frames of the first face image. Yet another example is capturing the first target object against the same background to obtain multiple frames of the first face image. For example, multiple frames of the first face images of the same first target object can be extracted from one or more existing publicly available datasets. The datasets used can be Voxceleb2, 300W-LP, and / or FEAFA, etc. Voxceleb2 is a dataset with large-scale face video data, 300W-LP is a dataset with rich face pose information, and FEAFA is a dataset with rich face expression information.

[0039] For example, the first face image in each frame is a face image with a known face identity. Here, face identity is a specific description of the face's shape, such as describing key facial features, including eyebrows, eyes, nose, mouth, and cheeks. For instance, the key features described by the face identity might be large eyes, a high nose bridge, and thin lips. Since different target objects have different faces, the key features described by the face identities of different target objects will vary. It is understandable that overlaying different expressions onto the face identity can yield faces with various expressions.

[0040] Step 120: Input each frame of the first face image into the corresponding coefficient regression network to obtain the face reconstruction coefficients of each frame of the first face image. Each coefficient regression network has the same network parameters. The face reconstruction coefficients include identity coefficients and expression coefficients.

[0041] The coefficient regression network is a neural network used to obtain the coefficients needed to reconstruct a face. The specific structure of the coefficient regression network can be set according to actual conditions. For example, a lightweight neural network such as MobileNet-V2 or other lightweight neural networks can be used as the network skeleton of the coefficient regression network to accelerate network inference and meet the real-time requirements of the application scenario. During training, the number of coefficient regression networks used is equal to the number of frames in the multiple first face images to ensure that each first face image has a corresponding coefficient regression network for processing. In one embodiment, multiple coefficient regression networks share network parameters, that is, each coefficient regression network has the same network parameters, where network parameters refer to parameters such as convolutional kernel weights used in the coefficient regression network. It can be understood that when constructing the coefficient regression network, default (e.g., manually set) network parameters are used, and the network parameters are continuously updated as the number of training iterations increases to improve the performance of the coefficient regression network and make the coefficient regression network output more accurate results.

[0042] For example, a first face image is input into a coefficient regression network. The coefficient regression network processes the first face image to output the coefficients needed to reconstruct the face in the first face image. In one embodiment, the coefficients output by the coefficient regression network are denoted as face reconstruction coefficients. The types of coefficients included in the face reconstruction coefficients can be selected according to the actual situation. In one embodiment, the face reconstruction coefficients include: identity coefficients and expression coefficients.

[0043] Identity coefficients describe the facial identity of the first input face image. Optionally, various identity bases that can appear on the face are pre-defined. Each identity base has a semantic meaning for a key feature, and each identity base can be considered as a category in which that key feature can appear. For example, when the key feature is the mouth, the corresponding identity bases include upper thick lip, upper thin lip, upper small (short) lip, upper large (long) lip, lower thick lip, lower thin lip, lower small (short) lip, upper large (long) lip, etc. Identity coefficients can reflect the weight (or confidence level) of each identity base appearing in the first face image. Identity coefficients can be in the form of a matrix vector, where each value corresponds to an identity base. The larger the value, the higher the probability that the identity base appears in the face. During the reconstruction process, based on the identity coefficients, each identity base is superimposed on the pre-built face model to reconstruct the facial identity of the first face image.

[0044] Expression coefficients are used to describe the facial expressions of the first input face image. Optionally, various expression bases that can appear on a face are pre-defined, with each expression base corresponding to one expression. It is understood that since facial expressions are very rich, representing all facial expressions would require constructing a large number of expression bases, which would significantly increase the amount of data processing. Therefore, in one embodiment, only basic expression bases are constructed. The specific content of the basic expressions can be set according to actual conditions. Various facial expressions can be obtained by combining and superimposing the basic expressions onto the face identity. For example, basic eye expressions include: left eye closed, left eye wide open, right eye closed, and right eye wide open. Based on these four basic expressions and the eyes in the face identity, various eye expressions can be obtained. For example, the expression of slightly squinting eyes can be obtained by linearly superimposing the left eye closed, right eye closed, and the eyes in the face identity. Expression coefficients can reflect the weight (or confidence level) of each expression base appearing in the first face image. Expression coefficients can be in the form of a matrix vector, where each value in the expression coefficient corresponds to an expression base. The larger the value, the higher the probability that the expression base appears on the face. Taking the eyes as an example, during the reconstruction process, based on the coefficients of each expression base obtained for the eyes, the corresponding expression base is applied to the eyes of the face identity (the larger the coefficient, the larger the proportion of the expression base), and the eyes with expressions in the first face image can be reconstructed.

[0045] It is understood that the above face reconstruction coefficients are only illustrative examples. In practical applications, the categories of face reconstruction coefficients can be increased or decreased according to actual needs.

[0046] It should be noted that each coefficient regression network outputs a corresponding face reconstruction coefficient, meaning that each frame of the first face image has a corresponding face reconstruction coefficient.

[0047] Step 130: Obtain the two-dimensional reconstructed image of the first face image in the corresponding frame based on the face reconstruction coefficients.

[0048] For example, a 3D face image of the face in the first face image is reconstructed based on the face reconstruction coefficients, and then mapped onto a 2D plane to obtain a 2D face image reconstructed from the first face image. In one embodiment, the obtained 2D face image is denoted as the 2D reconstructed image. It can be understood that the 2D reconstructed image can be considered as a face image reconstructed based on the face reconstruction coefficients. The more accurate the face reconstruction coefficients output by the coefficient regression network, the more similar the 2D reconstructed image is to the corresponding first face image.

[0049] In one embodiment, when the face reconstruction coefficients include identity coefficients and expression coefficients, the three-dimensional reconstructed image obtained through the face reconstruction coefficients can describe the face identity of the first target object and the expression appearing in the corresponding first face image.

[0050] For example, an average 3D face shape, identity base, and expression base are pre-constructed. The 3D face shape can be understood as a pre-constructed face model, which can be obtained by averaging a large number of 3D face images. The identity base can be considered as a set of pre-constructed identity bases, and the expression base can be considered as a set of pre-constructed expression bases. Then, the identity coefficients are applied to the identity bases, the expression coefficients are applied to the expression bases, and then they are superimposed with the average 3D face shape to obtain a 3D reconstructed image.

[0051] The 3D reconstructed image is then mapped onto a 2D plane to obtain a 2D reconstructed image. In one embodiment, a software renderer is used for image mapping. The specific type of software renderer is not currently limited.

[0052] Step 140: Construct a loss function for the coefficient regression network based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients. The loss function includes the unsupervised training loss function and the identity constraint loss function for each frame of the first face image.

[0053] For example, the higher the performance of the coefficient regression network, the more similar the 2D reconstructed image obtained based on the face reconstruction coefficients output by the coefficient regression network is to the first face image input to the coefficient regression network. Therefore, the degree of difference between the first face image and the corresponding 2D reconstructed image can be compared, and the network parameters of the coefficient regression network can be adjusted according to the degree of difference to improve the performance of the coefficient regression network. Based on this, to further improve the performance of the coefficient regression network, the network parameters are also adjusted using supervised training (i.e., using prior knowledge) in conjunction with the face reconstruction coefficients. For example, based on the known identity coefficients of the first face image and the identity coefficients output by the coefficient regression network, it can be determined whether the identity coefficients output by the coefficient regression network are accurate. The smaller the difference between the known identity coefficients and the identity coefficients, the more accurate the identity coefficients. Then, the network parameters of the coefficient regression network can be adjusted according to the degree of difference to improve the accuracy of the identity coefficients, thereby improving the performance of the coefficient regression network.

[0054] In one embodiment, before adjusting the network parameters of the coefficient regression network, a loss function for the coefficient regression network is constructed based on the first face image, the corresponding 2D reconstructed image, and the face reconstruction coefficients. The network parameters are then adjusted using this loss function. The loss function evaluates the degree to which the predicted values ​​of the model (here, the coefficient regression network) differ from the true values; a better loss function indicates better model performance. Generally, an expression for the loss function is pre-constructed. The variables in this expression are related to the face reconstruction coefficients. These variables can be relevant data from the face reconstruction coefficients or relevant data from the 2D reconstructed image obtained based on the face reconstruction coefficients. Substituting the variables obtained during each training process into the expression yields the loss function (here, the specific value of the loss function) for that training process. The network parameters are then adjusted based on this loss function. It is understood that the loss function gradually converges as the number of training iterations increases.

[0055] The type of loss function can be set according to the actual situation. The current loss function includes the unsupervised training loss function and the identity constraint loss function for the first face image of each frame.

[0056] For example, unsupervised training can also be understood as unsupervised learning, which can solve problems in the training process of a neural network based on samples without categories (i.e., unlabeled). In other words, no prior knowledge is used during the learning process. The unsupervised training loss function can be understood as the loss function used in the unsupervised training process. It can be constructed based on the first face image and the 2D reconstructed image, or it can be constructed based on the face reconstruction coefficients, and no prior knowledge is used in the construction process. In one embodiment, the unsupervised training loss function includes at least one of the following: an image reconstruction loss function for the first face image, an image perception loss function, a keypoint loss function, and a regularization loss function. The image reconstruction loss function reflects the pixel differences between the first face image and the corresponding 2D reconstructed image. In one embodiment, the difference between pixels with the same pixel index (the same pixel index corresponds to the same pixel position in both images) in the first face image and the 2D reconstructed image is calculated using the L2 norm (here, the difference in pixel values). Then, the image reconstruction loss function is obtained using the differences corresponding to each pixel index. It can be understood that the smaller the pixel difference, the smaller the image reconstruction loss function. The image perception loss function reflects the difference in image features between the first face image and the corresponding 2D reconstructed image. In one embodiment, a pre-trained face recognition network (used to extract face-related features from the image) is used to extract image features from the first face image and the 2D reconstructed image, respectively. Then, the similarity between the two image features is calculated, and the image perception loss function is obtained based on the similarity. It can be understood that the higher the similarity, the smaller the image perception loss function. The keypoint loss function reflects the positional difference of facial keypoints between the first face image and the corresponding 2D reconstructed image. In one embodiment, each facial keypoint in the first face image and the 2D reconstructed image is extracted. Facial keypoints refer to points located in key areas, and each facial keypoint represents semantic information of the face. The selection of facial keypoints can be set according to actual conditions. For example, currently, a face may have 68 facial keypoints, including 5 keypoints each for the left and right eyebrows, 6 keypoints each for the left and right eyes, 20 keypoints for the mouth, 17 keypoints for the cheeks, and 9 keypoints for the nose. Both the first face image and the corresponding 2D reconstructed image have 68 facial keypoints. Next, the positional differences between facial keypoints with the same semantic information in the first face image and the 2D reconstructed image are calculated, and a keypoint loss function is constructed based on these positional differences. It can be understood that the smaller the positional difference, the smaller the keypoint loss function. A regularized loss function is used to constrain the coefficient regression network to improve its performance. For example, to prevent the degradation of face shape in 3D face images, the L2 norm can be used to constrain the identity coefficients. Similarly, to prevent the sparsity of expression coefficients, the L1 norm can be used to constrain the expression coefficients.After obtaining the above loss functions, summing them together yields the unsupervised training loss function. During summation, corresponding hyperparameters can be set for each loss function based on actual needs to ensure a more reasonable unsupervised training loss function. Each frame of the first face image has a corresponding unsupervised training loss function.

[0057] In addition to constructing an unsupervised training loss function, an identity constraint loss function also needs to be constructed. To ensure that the coefficient regression network outputs accurate expression coefficients, it is necessary to guarantee that the coefficient regression network outputs accurate identity coefficients. That is, assuming the reconstructed 3D face image is accurate, only by fitting accurate identity coefficients (i.e., outputting accurate identity coefficients) can the fitting of accurate expression coefficients be guaranteed. For example, to ensure the accuracy of identity coefficients, an identity constraint loss function is designed to enable the coefficient regression network to predict consistent and accurate identity coefficients for different first face images of the same first target object. The identity constraint loss function can be understood as a loss function that constrains the identity coefficients. The identity constraint loss function uses prior information, i.e., each frame of the first face image currently used has known identity coefficients. Then, the identity constraint loss function is constructed based on the known identity coefficients and the identity coefficients output by the coefficient regression network. In one embodiment, the L2 norm is used to calculate the error between the known identity coefficients and the identity coefficients, and then the identity constraint loss function is obtained based on this error. Alternatively, the average coefficient of the known identity coefficients of each frame of the first face image is calculated, and the L2 norm is used to calculate the error between the average coefficient and the identity coefficients, and then the identity constraint loss function is obtained based on this error. Understandably, the smaller the difference between the known identity coefficients and the identity coefficients output by the coefficient regression network, the smaller the identity constraint loss function, and the more accurate the identity coefficients output by the coefficient regression network. Each frame of the first face image has a corresponding identity constraint loss function.

[0058] In one embodiment, all coefficient regression networks share the same loss function and update their network parameters based on this same loss function, ensuring that the network parameters are identical during training. The loss function of the coefficient regression network can consist of an unsupervised training loss function for each frame of the first face image and an identity constraint loss function. Hyperparameters can be set for the identity constraint loss function to improve its accuracy. In this case, the expression for the loss function of each coefficient regression network is:

[0059]

[0060] Where L represents the loss function of the coefficient regression network, T is the total number of frames of the first acquired face image, and t represents the t-th frame. Let represent the unsupervised training loss function corresponding to the first face image in frame t. Let λ represent the identity constraint loss function corresponding to the first face image in frame t. idc The hyperparameters of the identity constraint loss function are represented.

[0061] Step 150: Update the network parameters of the coefficient regression network according to the loss function.

[0062] For example, the network parameters of each coefficient regression network are updated according to the loss function of the coefficient regression network to improve its performance and make the output of more accurate face reconstruction coefficients. After the network parameters are updated, the training process is considered complete. Then, new multi-frame images of the first face (the first target object in these images can change) are acquired again, and the above process is repeated to obtain the loss function of the coefficient regression network and update its network parameters again, until the training stop condition is met. The training stop condition can be set according to the actual situation. For example, a total number of training iterations can be set, and after each training iteration, the currently recorded number of training iterations can be updated. Then, it is determined whether the currently recorded number of training iterations has reached the total number of training iterations. If it has, training ends. Otherwise, a new training iteration begins. The training stop condition can also be that the loss function converges. Loss function convergence means that the loss functions obtained from consecutive training iterations are all small and the differences between the loss functions are within a set range. When the training stop condition is met, the face reconstruction coefficients output by the coefficient regression network can be considered to meet the requirement of accurately reconstructing face images.

[0063] The above-described method addresses the technical problem of low accuracy in facial expression coefficient reconstruction when reconstructing 3D facial images. By acquiring multiple frames of a first face image containing the same target object and processing each frame of the first face image using coefficient regression networks with identical network parameters to obtain face reconstruction coefficients, a 2D reconstructed image corresponding to the first face image is reconstructed based on these coefficients. An unsupervised training loss function and an identity constraint loss function are constructed based on the first face image, the 2D reconstructed image, and the face reconstruction coefficients. The network parameters of the coefficient regression network are then updated based on these two loss functions. By having the coefficient regression network output identity coefficients and facial expression coefficients separately, the identity coefficients and facial expression coefficients can be determined independently. Furthermore, the identity constraint loss function ensures the accuracy of the identity coefficients, meaning that the coefficient regression network can obtain the same facial identity based on different images of the same person, regardless of facial expression, thus guaranteeing the accuracy of the facial expression coefficients. Meanwhile, in addition to constraining the accuracy of the identity coefficients, an unsupervised training loss function was constructed. When comparing the first face image and the 2D reconstructed image, three dimensions were considered: pixel difference, feature difference, and key point difference. When using face reconstruction coefficients, constraints were imposed on the identity coefficients and expression coefficients respectively, which improved the rationality of the loss function. As a result, when training the coefficient sharing network based on the loss function, the performance of the coefficient sharing network was improved, further ensuring the accuracy of the expression coefficients.

[0064] Figure 2 This is a flowchart of a network training method for face reconstruction provided in one embodiment of this application. Figure 2 The method shown is a specific implementation of the method described above.

[0065] Figure 2 In the method shown, to improve the accuracy of 3D face image reconstruction, the content of face reconstruction coefficients is enriched. For example, in addition to expression coefficients and identity coefficients, face reconstruction coefficients can also include texture coefficients and pose coefficients. The texture coefficients can reconstruct the skin texture in the 3D face image, and the pose coefficients can reconstruct the head movement posture in the 3D face image. Thus, when obtaining a 2D reconstructed image based on the 3D face image, the 2D reconstructed image is made closer to the real first face image. This is used to calculate the loss function, making the loss function more accurately express the difference between the first face image and the 2D reconstructed image.

[0066] refer to Figure 2 The method specifically includes:

[0067] Step 210: Obtain multiple frames of the first face image of the first target object.

[0068] Step 220: Input each frame of the first face image into the corresponding coefficient regression network to obtain the face reconstruction coefficients of each frame of the first face image. Each coefficient regression network has the same network parameters. The face reconstruction coefficients include identity coefficients, expression coefficients, texture coefficients, and pose coefficients.

[0069] In addition to identity coefficients and expression coefficients, face reconstruction coefficients also include texture coefficients and pose coefficients.

[0070] Texture coefficients describe the skin texture of the first input face image. Optionally, similar to identity and expression coefficients, pre-defined texture bases are used to represent textures that can appear on the face's skin surface. Each texture base represents a type of texture that can appear in facial expressions. Texture coefficients reflect the weights of the skin surface of the first face image belonging to each texture base. Texture coefficients can be in matrix-vector form, where each value corresponds to a texture base; a larger value indicates a higher probability of that texture base appearing on the face. During reconstruction, based on the texture coefficients, the corresponding texture bases are applied to the pre-defined texture model to reconstruct the skin texture of the first face image.

[0071] Pose coefficients are used to describe the pose of the face's head in the current input first face image, where the pose of the first face image includes the face's position and orientation. During the reconstruction process, the pose coefficients are applied to the reconstructed 3D face image to reconstruct the pose of the face in the first face image.

[0072] It is understood that the above face reconstruction coefficients are merely illustrative examples. In practical applications, the categories of face reconstruction coefficients can be increased or decreased according to actual needs. For example, in practical applications, when it is not necessary to reconstruct skin texture, texture coefficients can be deleted. As another example, in practical applications, illumination coefficients describing the lighting in the first input face image can be added to the face reconstruction coefficients to simulate the corresponding lighting when reconstructing the face image.

[0073] Step 230: Obtain the two-dimensional reconstructed image of the first face image in the corresponding frame based on the face reconstruction coefficients.

[0074] For example, when reconstructing the first target object based on face reconstruction coefficients, not only identity coefficients and expression coefficients are included, but also texture coefficients and pose coefficients. In one embodiment, the 3D face image reconstructed based on identity coefficients and expression coefficients can reflect the 3D shape of the face (i.e., the surface shape of the face). Therefore, the 3D face image reconstructed based on identity coefficients and expression coefficients is denoted as the 3D face shape, meaning the 3D face shape can describe the face identity of the first target object and the expression appearing in the corresponding first face image. The 3D texture of the face can be reconstructed based on texture coefficients. Therefore, the 3D face image reconstructed based on texture coefficients is denoted as the 3D face texture, meaning the 3D face texture can describe the skin texture of the first target object. It can be understood that the corresponding 3D face shape and 3D face texture can be reconstructed using the face reconstruction coefficients corresponding to each frame of the first face image.

[0075] Subsequently, based on the 3D face shape and texture, a 2D reconstructed image containing texture, identity, and expression can be obtained. Pose coefficients are then applied to this 2D reconstructed image to include the head movements of the first face image. For example, after obtaining the 3D face shape and texture, they are mapped onto a 2D plane. This involves rendering the textured 3D face (i.e., the 3D face shape and texture) onto the image plane (i.e., the 2D plane) of the first face image, thus obtaining a 2D reconstructed image representing the first target object. Each frame of the first face image corresponds to one 2D reconstructed image. During mapping, pose coefficients can also be introduced to give the face in the 2D face image a specific pose.

[0076] When obtaining a two-dimensional reconstructed image based on the three-dimensional face shape and texture, step 230 includes steps 231-232:

[0077] Step 231: Reconstruct the 3D face shape of the first face image in the corresponding frame based on the identity coefficient and expression coefficient, and reconstruct the 3D face texture of the first face image in the corresponding frame based on the texture coefficient.

[0078] For example, the face of a first target object is reconstructed based on face reconstruction coefficients. In one embodiment, the reconstructed content includes a three-dimensional face shape and a three-dimensional face texture.

[0079] In one embodiment, the three-dimensional face shape can be obtained by superimposing an identity base and an expression base. In this case, step 231, reconstructing the three-dimensional face shape of the corresponding frame's first face image based on the identity coefficients and expression coefficients, specifically includes: applying the identity coefficients and expression coefficients to the corresponding identity base and expression base respectively, and combining them with the pre-constructed average face shape to obtain the three-dimensional face shape of the corresponding frame's first face image.

[0080] The shape of a three-dimensional human face can be represented as:

[0081]

[0082] Where S(α,β) represents the reconstructed 3D face shape, B represents the pre-constructed average face shape. id B represents the identity basis. exp Let α represent the expression base, β represent the identity coefficient, and β represent the expression coefficient. The parameters corresponding to each of the above symbols can be represented in the form of a matrix vector.

[0083] Optionally, the process of constructing the average face shape can be as follows: the average coordinates (specifically, three-dimensional coordinates) of a certain number (e.g., all, 200, or 100) of three-dimensional faces in a currently publicly available database (such as the Basel FaceMode (BFM) database) are determined as the average face shape. This average face shape is a three-dimensional face shape, which can be considered as a pre-constructed face model.

[0084] An identity basis is a set of identity bases. Optionally, the construction process of the identity bases involves using a publicly available database (such as the BFM database) and obtaining each identity base based on Principal Component Analysis (PCA). In one embodiment, 80 identity bases were constructed in the BFM database based on PCA. Each identity base has corresponding coefficients in the identity coefficients.

[0085] An expression base is a collection of expression templates. Optionally, the expression base is constructed as follows: Based on the FaceWarehouse dataset (a dataset containing facial expressions), the Facial Action Coding System (FACS) is used to classify each basic expression by muscle movements to obtain expression templates representing each basic expression. Each expression template may include the basic expression it represents and a refined value indicating the amplitude of muscle movement. For example, an expression template may be AU1(0.2), where AU1 represents the basic expression of the inner eyebrow being pulled upwards, 0.2 represents the degree of muscle pulling, and the muscle pulling amplitude in expression template AU1(0.2) is smaller than that in expression template AU1(0.3). In one embodiment, 46 expression templates were constructed using FACS. Then, the expression templates are transferred to make them consistent with the topological structure of the 3D faces in the BFM database. The transferred expression templates are the expression bases. Transfer can be understood as deformation transfer, which is the transfer of a 3D face from one topological structure to a 3D face with another topological structure. The migration process can be as follows: Register 3D faces from the FaceWarehouse dataset (denoted as source 3D faces) and 3D faces from the BFM database (denoted as target 3D faces) to establish vertex associations between the source and target 3D faces. The vertex positions can be selected according to the actual situation, such as dividing the 3D face into multiple triangular patches, where the vertices of each triangular patch are the vertices to be associated. Then, solve the deformation variables between each expression template corresponding to the source 3D face and the face identity of the source 3D face. These deformation variables can be represented by an affine transformation matrix. Finally, through vertex association, use the deformation variables to deform the face identity of the target 3D face to obtain each expression basis. It should be noted that when reconstructing 3D face shapes, it is necessary to transfer expression bases with explicit semantic information, such as the expression base representing "closed eyes" or "frowning." However, expression bases obtained from publicly available databases (such as the BFM database) based on Principal Component Analysis (PCA) do not possess explicit semantic information. Therefore, the FaceWarehouse dataset, which contains explicit facial expressions, is used to construct the expression bases. Each expression base in the expression base has a corresponding coefficient found in the expression coefficients.

[0086] For example, after the coefficient regression network outputs identity coefficients and expression coefficients, substituting these coefficients into the above formula yields the 3D face shape. This formula can also be considered a model for reconstructing the 3D face shape; this model is linear.

[0087] In one embodiment, when reconstructing the 3D face texture, an average 3D face texture and a texture base are pre-constructed. Then, texture coefficients are applied to the texture base and superimposed with the average 3D face texture to obtain the 3D face texture. In this case, step 231, reconstructing the 3D face texture of the corresponding frame's first face image based on the texture coefficients, specifically includes: applying the texture coefficients to the corresponding texture base and combining them with the pre-constructed average face texture to obtain the 3D face texture of the corresponding frame's first face image.

[0088] A 3D human face texture can be represented as:

[0089]

[0090] Where T(δ) represents the reconstructed 3D face texture, B represents the pre-constructed average face texture. tex δ represents the texture base, and δ represents the texture coefficient. The parameters corresponding to each of the above symbols can be represented in the form of a matrix vector.

[0091] The construction process of average face texture is similar to that of average face shape. The construction process of average face texture is as follows: the average texture data of a certain number (e.g., all, 200 or 100) of three-dimensional faces in the currently publicly available database (such as the Basel Face Mode (BFM) database) is determined as the average face texture.

[0092] A texture base is a collection of texture bases. The construction process of texture bases is similar to that of identity bases. The process involves using a publicly available database (such as the BFM database) and obtaining the texture bases based on Principal Component Analysis (PCA). In one embodiment, 80 texture bases were constructed in the BFM database based on PCA. Each texture base has corresponding coefficients in its texture coefficients.

[0093] For example, after the coefficient regression network outputs texture coefficients, substituting these coefficients into the above formula yields the 3D face texture. This formula can also be considered a model for reconstructing the 3D face texture; this model is linear.

[0094] It is understood that the above reconstruction methods for 3D face shape and 3D face texture are merely illustrative descriptions. In practical applications, other reconstruction methods may also be used, which are not currently limited.

[0095] Each frame of the first face image corresponds to a reconstructed 3D face shape and 3D face texture. After obtaining the 3D face shape and 3D face texture, step 232 is executed.

[0096] Step 232: Render the two-dimensional reconstructed image of the first face image of the corresponding frame based on the pose coefficient, three-dimensional face shape and three-dimensional face texture.

[0097] The 3D face shape and texture are mapped onto a 2D plane, and pose coefficients are added during the mapping process to obtain a 2D reconstructed image.

[0098] In one embodiment, a renderer is used to render a two-dimensional reconstructed image. In this case, step 232 includes steps 2321-2322:

[0099] Step 2321: Apply the pose coefficients to the perspective projection model.

[0100] Perspective projection is a method for drawing or rendering on a two-dimensional plane to achieve a visual effect close to that of a realistic three-dimensional object. A perspective projection model refers to a model that uses perspective projection. This model can be a perspective projection camera model, which can render a two-dimensional image using the camera's intrinsic and extrinsic parameters. Intrinsic parameters refer to the camera's internal parameters, such as the camera's focal length and the coordinates of the center point of the first face image. Extrinsic parameters refer to the camera's external parameters; currently, extrinsic parameters refer to the pose coefficients output by the coefficient regression network, which are applied to the perspective projection model. Optionally, the number of perspective projection models can be the same as the number of frames of the first face image, so that the pose coefficients corresponding to each frame of the first face image are applied to the corresponding perspective projection model.

[0101] Step 2322: Input the perspective projection model, 3D face shape and 3D face texture into the renderer, and render the 2D reconstructed image of the first face image of the corresponding frame.

[0102] For example, the renderer is a software renderer. In one embodiment, the renderer is a differentiable renderer. Differentiable rendering can be understood as calculating the derivative of the rendering process, thereby incorporating the rendering process into a neural network to solve more complex visual problems. In one embodiment, a two-dimensional reconstructed image can be reconstructed using a differentiable renderer. It is understood that there can be one or more renderers. When there are multiple renderers, the number of renderers can be equal to the number of frames in the first face image. In this case, each renderer renders one two-dimensional reconstructed image. When there is only one renderer, the renderer sequentially reconstructs the two-dimensional reconstructed images corresponding to each first face image.

[0103] In one embodiment, a perspective projection model with applied pose coefficients, a 3D face shape, and a 3D face texture are used together as input to the renderer, and the renderer processes these elements to output a 2D reconstructed image. For example, during the renderer's operation, the 3D face shape is projected onto a 2D plane to obtain a 2D image, and the 3D face texture is rendered onto the 2D image. During both projection and rendering, a perspective projection model with applied pose coefficients is used to obtain a 2D reconstructed image that includes head movement poses.

[0104] In one embodiment, to improve the performance of the coefficient regression network, i.e., to improve the accuracy of the 2D reconstructed image, an illumination system is added to the face reconstruction coefficients. The illumination coefficients reflect the ambient lighting during the acquisition of the first face image, thereby enhancing the 3D face texture. Optionally, various lighting categories appearing during image acquisition are pre-set. The illumination coefficients reflect the weight of the ambient lighting of the first face image belonging to each lighting category; the larger the value of a certain lighting category in the illumination coefficients, the higher the probability that it appears in the first face image. Based on this, the face reconstruction coefficients also include: illumination coefficients; before step 232, the method further includes: applying the illumination coefficients to the illumination model; inputting the 3D face texture into the illumination model, and adding the ambient lighting of the first face image to the 3D face texture through the illumination model.

[0105] A lighting model is a computer model that simulates the physical process of light illumination in nature based on the laws of optics. The type of lighting model used can be set according to actual conditions. In one embodiment, a spherical harmonic lighting model is used as an example. The spherical harmonic lighting model is used for 3D face textures to add the ambient lighting of the first face image captured, thereby enhancing skin texture. In this case, the lighting coefficients output by the coefficient regression network are used as the coefficients of the spherical harmonic lighting model. Then, the spherical harmonic lighting model is applied to process the 3D face texture. During processing, the spherical harmonic lighting model simulates the lighting environment according to the lighting coefficients to add the ambient lighting of the first face image to the 3D face texture. It can be understood that there can be one or more lighting models. When there are multiple lighting models, the number of lighting models can be equal to the number of frames of the first face image. In this case, each lighting model renders one 2D reconstructed image. When there is only one lighting model, the lighting model processes the 3D face textures corresponding to each first face image sequentially.

[0106] It should be noted that when applying the lighting model, the 3D face texture input to the renderer when reconstructing the 2D image is a 3D face texture processed by the lighting coefficients and lighting model.

[0107] After obtaining the two-dimensional reconstructed image, proceed to step 240.

[0108] Step 240: Construct the loss function of the coefficient regression network based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients. The loss function includes the unsupervised training loss function and the identity constraint loss function corresponding to each frame of the first face image.

[0109] For example, the currently used 2D reconstructed image is an image with added texture and pose, or it could be an image with added texture, pose, and lighting.

[0110] The loss function constructed based on the first face image, the corresponding 2D reconstructed image, and face reconstruction coefficients includes an unsupervised training loss function and an identity constraint loss function for each frame of the first face image. The unsupervised training loss function includes at least one of the following: an image reconstruction loss function for the first face image, an image perception loss function, a keypoint loss function, and a regularization loss function.

[0111] In one embodiment, the unsupervised training loss function includes: an image reconstruction loss function for the first face image. In this case, constructing the unsupervised training loss function based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients includes step 241:

[0112] Step 241: Calculate the pixel error between the first face image and the corresponding two-dimensional reconstructed image using the L2 norm, and obtain the image reconstruction loss function of the first face image based on the pixel error.

[0113] It is understandable that the more accurate the face reconstruction coefficients, the more similar the pixels at the same position (i.e., the same coordinates) between the first face image and the corresponding 2D reconstructed image. Therefore, an image reconstruction loss function can be constructed based on the pixel error between the first face image and the 2D reconstructed image. In one embodiment, the L2 norm is used when constructing the image reconstruction loss function. In this case, the expression for the image reconstruction loss function is:

[0114]

[0115] Among them, L pho This represents the image reconstruction loss function. This represents the region where the face is located in the 2D reconstructed image, that is, the projection area of ​​the face when the renderer renders the 2D reconstructed image. Each pixel in the face region can be clearly identified, where i represents the pixel index. Through i, you can The corresponding pixel was found in A. i This represents the mask value of the pixel with pixel index i in the face mask. The face mask is detected in the first face image using a face segmentation algorithm. The face mask represents the pixels displaying the face in the first face image. When A... iWhen the pixel corresponding to the first face image (i.e., the pixel with pixel index i in the first face image) represents the face, A i If it is 1, then A is the opposite. i It is 0. i This represents the pixel value of the pixel with pixel index i in the first face image. This represents the pixel value of the pixel with pixel index i in the 2D reconstructed image. Let ||.||2 represent the difference in pixel value of the pixel with pixel index i in the first face image and the 2D reconstructed image, where ||.||2 represents the L2 norm.

[0116] It is understandable that after obtaining the 2D reconstructed image corresponding to the first face image each time, L can be obtained based on the above expression. pho The specific value of . During a training process, each frame of the first face image corresponds to an image reconstruction loss function.

[0117] In one embodiment, the unsupervised training loss function includes: an image perception loss function for the first face image. In this case, constructing the unsupervised training loss function based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients includes steps 242-243:

[0118] Step 242: Use a face recognition network to extract the first image features of the first face image and the second image features of the corresponding two-dimensional reconstructed image.

[0119] A face recognition network is a pre-trained neural network, such as a convolutional neural network or a residual neural network, used to extract features from input images. When a first face image is input into the face recognition network, it can extract depth features related to the face, denoted as the first image feature. Similarly, when the corresponding 2D reconstructed image is input into the face recognition network, it can extract depth features related to the face from the 2D reconstructed image, denoted as the second image feature.

[0120] Step 243: Calculate the similarity between the first image features and the second image features, and obtain the image perception loss function of the first face image based on the similarity.

[0121] The more similar the first face image and its corresponding 2D reconstructed image are, the more similar the first image features and second image features extracted based on the same face recognition network are. Therefore, by calculating the similarity between the first image features and the second image features, the degree of similarity between the first face image and the 2D reconstructed image can be determined. Based on this, an image perception loss function is designed to reflect the degree of similarity between the first image features and the second image features. It can be understood that, for this training process, each frame of the first face image has an image perception loss function.

[0122] In one embodiment, cosine similarity is used to calculate the similarity between the first image feature and the second image feature, thereby obtaining the image perception loss function. The higher the similarity, the smaller the image perception loss function. For example, the expression for the image perception loss function is:

[0123]

[0124] Among them, L per Let f(I) represent the image perception loss function. i () represents the first image feature of the first face image. The second image feature represents the two-dimensional reconstructed image. f(I) i )and The inner product, ||f(I) i || represents f(I) i ) length, express The length of L. It can be understood that after obtaining the 2D reconstructed image corresponding to the first face image each time, L can be obtained based on the above expression. pr The specific value of e.

[0125] In one embodiment, the unsupervised training loss function includes a keypoint loss function for the first face image. In this case, constructing the unsupervised training loss function based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients includes steps 244-245:

[0126] Step 244: Obtain the first facial key points corresponding to the first facial image and the second facial key points of the corresponding two-dimensional reconstructed image.

[0127] For example, facial key points can be used to determine the movements of key parts. In one embodiment, facial key points in the first face image are designated as first facial key points, and facial key points in the 2D reconstructed image are designated as second facial key points. Optionally, for ease of calculation, a key point index is set for each facial key point, and facial key points with the same semantic information in the first face image and the 2D reconstructed image have the same key point index, such as the leftmost facial key point of the left eyebrow having the same key point index.

[0128] The method for determining facial landmarks is not currently limited. For example, for a first face image, a face recognition algorithm is used to detect the location region of the first target object's face in the first face image. Then, facial landmark detection technology is used to detect the first facial landmarks in the location region. As another example, for a 2D reconstructed image, after obtaining the 3D face shape, facial landmark detection technology can be used to detect 3D facial landmarks in the 3D face shape. Then, when the renderer projects the 3D face shape onto a 2D plane, it can project the 3D facial landmarks onto the 2D plane to obtain the second facial landmarks of the 2D reconstructed image.

[0129] Step 245: Calculate the mean square error of the position between the first and second facial key points, and obtain the key point loss function of the first facial image based on the mean square error.

[0130] The more similar the first face image is to its corresponding 2D reconstructed image, the more similar the positions of the first facial keypoints with the same semantic information in the first face image are to the positions of the second facial keypoints in the 2D reconstructed image. Therefore, the similarity between the first face image and the 2D reconstructed image can be determined by calculating the positional difference between the first and second facial keypoints. Based on this, a keypoint loss function is designed to reflect the degree of positional difference between the first and second facial keypoints. It can be understood that for this training process, each frame of the first face image has a keypoint loss function.

[0131] In one embodiment, the positional difference between a first facial keypoint and a second facial keypoint with the same semantics is calculated using mean squared error, thereby obtaining the keypoint loss function. The smaller the positional difference, the smaller the keypoint loss function. For example, the expression for the keypoint loss function is:

[0132]

[0133] Among them, L lm Let n represent the keypoint loss function, n represent the total number of facial keypoints (i.e., the maximum keypoint index), i represent the current keypoint index, and ω represent the maximum keypoint index.i This represents the weight of the facial keypoint with keypoint index i. The weight can be preset according to actual needs. For example, the weight of the facial keypoint on the cheek can be set to 1, and the weight of the other facial keypoints can be set to 10, so as to focus on the eyes, nose, mouth and eyebrows that are related to facial identity and expression. Q i This indicates the position (i.e., coordinates) of the first facial keypoint with keypoint index i in the first face image. This represents the position (coordinates) of the second facial keypoint with keypoint index i in the 2D reconstructed image. 2 This represents the square of the L2 norm. It can be understood that after obtaining the 2D reconstructed image corresponding to the first face image, L can be obtained based on the above function formula. lm The specific value.

[0134] In one embodiment, the unsupervised training loss function includes: a regularization loss of the first face image. In this case, constructing the unsupervised training loss function based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients includes steps 246-248:

[0135] Step 246: Calculate the first norm square of the identity coefficient and the second norm square of the texture coefficient of the first face image, and obtain the identity and texture constraint loss function of the first face image based on the first norm square and the second norm square.

[0136] To prevent degradation of 3D face shape and texture, constraints are imposed on the identity and texture coefficients in the face reconstruction coefficients to construct a corresponding loss function. This loss function is denoted as the identity and texture constraint loss function. In one embodiment, the expression for the identity and texture constraint loss function is:

[0137]

[0138] Among them, L reg Let M represent the identity and texture constraint loss function, where M is the total number of identity and texture bases (i.e., the total number of categories). In one embodiment, the total number of identity and texture bases is the same, both being 80, i.e., M = 80. n represents the index of the texture base and the index of the identity base, and α... n δ represents the coefficient corresponding to the identity basis with index n in the identity coefficients. n This represents the coefficient corresponding to the texture base with index n in the texture coefficients. 2 λ represents the square of the L2 norm. α and λ δ These are all hyperparameters, and their specific values ​​can be set according to the actual situation. For example, they are currently set to 1 and 0.001 respectively. It can be understood that after obtaining the face reconstruction coefficients corresponding to the first face image each time, L can be obtained based on the above representation.reg The specific value. Each frame's first face image corresponds to an identity and texture constraint loss function.

[0139] Step 247: Use the L1 norm to constrain the expression coefficients of the first face image to obtain the expression constraint loss function of the first face image.

[0140] To prevent sparsity of expression coefficients, L1 regularization is applied to constrain the expression coefficients in the face reconstruction coefficients to construct a corresponding loss function. This loss function is denoted as the expression constraint loss function. In one embodiment, the expression constraint loss function is expressed as follows:

[0141]

[0142] Among them, L sp Let represent the expression constraint loss function, where m is the total number of expression bases (i.e., the total number of categories). In one embodiment, there are 46 expression bases, i.e., m = 46. i represents the index of the expression base, and β represents the coefficient of the expression base with index i in the expression coefficients. |.| represents the L1 norm. λ α This is a hyperparameter, and its specific value can be set according to the actual situation, for example, λ. α The value can be related to λ in the above identity and texture constraint loss function. α The values ​​may be the same or different. It can be understood that after obtaining the face reconstruction coefficients corresponding to the first face image each time, the above expression can be used to calculate L. sp The specific value. Each frame's first face image corresponds to an expression constraint loss function.

[0143] It is understandable that steps 246 and 247 can be executed simultaneously or sequentially, and their specific execution order is not currently limited.

[0144] Step 248: Obtain the regularization loss function of the first face image based on the identity and texture constraint loss function and the expression constraint loss function.

[0145] For example, the regularization loss consists of an identity and texture loss function and an expression constraint loss function. In one embodiment, corresponding hyperparameters can be set for the identity and texture loss function and the expression constraint loss function, respectively. The specific values ​​of the hyperparameters can be set according to the actual situation and are not limited here.

[0146] It is understandable that when the unsupervised training loss function includes multiple loss functions such as image reconstruction loss function, image perception loss function, keypoint loss function, and regularization loss function for the first face image, these loss functions can be calculated simultaneously or in a predetermined order; this is not currently restricted. Furthermore, corresponding hyperparameters can be set for each loss function to obtain the final unsupervised training loss function. For example, the unsupervised training loss function includes image reconstruction loss function, image perception loss function, keypoint loss function, and regularization loss function for the first face image. In this case, the unsupervised training loss function can be expressed as:

[0147] L com =λ pho L pho +λ per L per +λ lm L lm +λ reg L reg +λ sp L sp

[0148] Among them, L com L represents the unsupervised training loss function. pho Let λ represent the image reconstruction loss function. pho For L pho hyperparameters, L per Let λ represent the image perception loss function. per For L per hyperparameters, L lm Let λ represent the keypoint loss function. lm For L lm hyperparameters, L reg The loss function representing identity and texture constraints, λ reg For L reg hyperparameters, L sp Let λ represent the expression constraint loss function. sp For L sp hyperparameters, L reg and L sp This forms the regularized loss function. The specific values ​​of each hyperparameter can be set based on the importance of the corresponding loss function and the training results of the coefficient regression network. For example, setting λ... pho It is 1.9, λ per λ is 0.2 lm λ is 0.1 reg λ is 0.0001. sp It is 0.1.

[0149] The unsupervised training loss function corresponding to the first face image in each frame can be obtained using the above formula.

[0150] In addition to constructing the unsupervised training loss function, an identity constraint loss function also needs to be constructed. To ensure that the coefficient regression network outputs accurate expression coefficients, it is necessary to guarantee that the coefficient regression network outputs accurate identity coefficients. That is, assuming the reconstructed 3D face shape is accurate, only by fitting accurate identity coefficients (i.e., outputting accurate identity coefficients) can the fitting of accurate expression coefficients be guaranteed. In one embodiment, constructing the identity constraint loss function of the coefficient regression network based on the first face image, the corresponding 2D reconstructed image, and the face reconstruction coefficients includes steps 249-2411:

[0151] Step 249: Obtain the identity label coefficients corresponding to the first face image in each frame.

[0152] For example, the identity label coefficient of the first face image refers to the known identity coefficient of the first face image, which can be used as prior information. That is, when acquiring the first face image, the identity label coefficient of the first face image is also determined simultaneously.

[0153] Step 2410: Use the average coefficient of each identity label coefficient as the pseudo-label supervision coefficient.

[0154] In one embodiment, the average coefficient of the identity label coefficients for each frame of the first face image is calculated, and this average coefficient is used as a pseudo-label to supervise the coefficient regression network in outputting accurate identity coefficients. Currently, the average coefficient is denoted as the pseudo-label supervision coefficient. Optionally, for the first target object, the pseudo-label supervision coefficient reduces the changes in face identity caused by different expressions in multiple frames of the first face image; therefore, the accuracy of the pseudo-label supervision coefficient is higher than that of the identity label coefficient for a single frame of the first face image. In practical applications, the identity label coefficient of each first face image can also be used to supervise the corresponding coefficient regression network in outputting accurate identity coefficients.

[0155] Step 2411: Calculate the mean square error between the identity coefficient and the pseudo-label supervision coefficient of the first face image, and obtain the identity constraint loss function based on the mean square error.

[0156] In one embodiment, the identity constraint loss function for a first face image frame is defined as the mean square error between the pseudo-label supervision coefficients and the identity coefficients corresponding to the first face image in that frame. In this case, the expression for the identity constraint loss function is:

[0157]

[0158] in, Let α represent the identity constraint loss function for the first face image in frame t. t The identity coefficients corresponding to the first face image in frame t are represented by a matrix vector. This represents the pseudo-label supervision coefficients, which are in matrix-vector form. ||.||2 represents the square of the L2 norm. It can be understood that the more accurate the identity coefficients, the better. The smaller the difference between them.

[0159] After the coefficient regression network outputs the identity coefficients, the following expression can be used to obtain the identity coefficients: The specific value of . Furthermore, each frame of the first face image has a corresponding identity constraint loss function.

[0160] In one embodiment, after obtaining the identity constraint loss function and unsupervised training loss function of the first face image in each frame, the loss function of each coefficient regression network can be constructed, and the loss function of each coefficient regression network is the same.

[0161] Step 250: Update the network parameters of the coefficient regression network according to the loss function.

[0162] The above-described method addresses the problem of low accuracy in facial expression coefficient reconstruction when reconstructing 3D facial images. It involves acquiring multiple frames of a first face image containing the same target object and processing each frame using coefficient regression networks with identical network parameters to obtain face reconstruction coefficients. Then, a 2D reconstructed image corresponding to the first face image is reconstructed based on these coefficients. An unsupervised training loss function and an identity constraint loss function are constructed based on the first face image, the 2D reconstructed image, and the face reconstruction coefficients. The network parameters of the coefficient regression network are then updated based on these loss functions. By outputting identity coefficients, facial expression coefficients, texture coefficients, and pose coefficients through the coefficient regression network, a more accurate 2D reconstructed image is obtained. This makes the pixel differences, feature differences, and keypoint differences between the 2D reconstructed image and the first face image more accurate, improving the accuracy of the unsupervised training loss function. Furthermore, the identity constraint loss function ensures the accuracy of the identity coefficients, thereby guaranteeing the accuracy of the facial expression coefficients.

[0163] Figure 3 This is a flowchart of a network training method for face reconstruction provided in one embodiment of this application. Figure 3 The method shown is a specific implementation of the method described above.

[0164] It is understandable that some facial expressions cannot appear on a person's face simultaneously. For example, the expression base representing a closed left eye and the expression base representing a wide-open left eye cannot appear on a person's face at the same time. These two expression bases that cannot appear at the same time can be denoted as mutually exclusive expression pairs. The expression coefficients output by the coefficient regression network include the coefficients of the two expression bases in the mutually exclusive expression pair. That is, both expression bases are in an active state during the prediction process of the coefficient regression network. At this time, in order to avoid the influence of mutually exclusive expression pairs on the accuracy of expression coefficients, a loss function related to the mutually exclusive expression pairs can be constructed based on the expression coefficients output by the coefficient regression network. This will prevent the expression bases contained in the mutually exclusive expression pairs from appearing at the same time, improve the accuracy of expression coefficients, and thus improve the performance of the coefficient regression network.

[0165] Based on this, refer to Figure 3 The network training method for face reconstruction includes:

[0166] Step 310: Obtain multiple frames of the first face image of the first target object.

[0167] Step 320: Input each frame of the first face image into the corresponding coefficient regression network to obtain the face reconstruction coefficients of each frame of the first face image. Each coefficient regression network has the same network parameters. The face reconstruction coefficients include: identity coefficient, expression coefficient, texture coefficient, pose coefficient and illumination coefficient.

[0168] Step 330: Obtain the two-dimensional reconstructed image of the first face image in the corresponding frame based on the face reconstruction coefficients.

[0169] Step 340: Construct the loss function of the coefficient regression network based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients. The loss function includes the unsupervised training loss function of each frame of the first face image, the identity constraint loss function, and the expression mutual exclusion constraint loss function.

[0170] The unsupervised training loss function and the identity constraint loss function can be found in the aforementioned content.

[0171] For example, among the constructed expression bases, some are mutually exclusive and cannot appear simultaneously in a face. However, during the training of the coefficient regression network, since there is no direct supervision information for the expression coefficients (i.e., the expression bases appearing in the first face image are unknown beforehand), mutually exclusive expression bases will inevitably be predicted as active states by the coefficient regression network simultaneously. That is, the expression coefficients simultaneously contain coefficients corresponding to mutually exclusive expression bases, which hinders the accurate fitting of the expression coefficients and affects their accuracy. Therefore, the loss function of the coefficient regression network also includes: a loss function for the mutual exclusion constraint of expressions in each frame of the first face image, to guide the coefficient regression network to suppress mutually exclusive expression sets that should not appear simultaneously.

[0172] In one embodiment, when constructing the expression mutual exclusion constraint loss function, the construction of the expression mutual exclusion constraint loss function based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients includes steps 341-343:

[0173] Step 341: Obtain all pre-built mutually exclusive emoji pairs, each containing at least two mutually exclusive emoji bases.

[0174] For example, mutually exclusive expression bases are denoted as mutually exclusive expression pairs, each containing at least two mutually exclusive expression bases that cannot appear on a human face simultaneously. In one embodiment, an example is described where a mutually exclusive expression pair contains two mutually exclusive expression bases. Currently, all mutually exclusive expression pairs are pre-identified within the constructed expression bases. For example, Figure 4 This is a schematic diagram of a mutually exclusive emoji pair provided in one embodiment of this application. (Reference) Figure 4 The diagram shows a subset (6 groups in total) of mutually exclusive expression pairs found. Taking the top-left mutually exclusive expression pair as an example, it shows that the expression base representing the right eye closing and the expression base representing the right eye widening are a mutually exclusive expression pair. Here, B1 represents the expression base with expression base index 1, and B9 represents the expression base with expression base index 9. It can be understood that before training the coefficient regression network, all mutually exclusive expression pairs are searched first. During training, the searched mutually exclusive expression pairs can be directly used without repeated searches.

[0175] Step 342: Find the expression coefficients of the expression basis contained in each mutually exclusive expression pair in the first face image, and obtain the expression mutual exclusion constraint loss function of the first face image.

[0176] For example, after the coefficient regression network outputs the expression coefficients, the coefficients corresponding to all expression bases in the mutually exclusive expression pairs can be found in the expression coefficients. For example, if the mutually exclusive expression pairs contain expression bases B1 and B9, the coefficients corresponding to B1 and B9 can be found in the expression coefficients.

[0177] Step 343: Based on the found expression coefficients, use the exponential function to obtain the expression mutual exclusion constraint loss function of the first face image.

[0178] For example, an expression mutual exclusion constraint loss function is constructed based on the coefficients of each found mutually exclusive expression pair. In one embodiment, an indicator function is used to construct the expression mutual exclusion constraint loss function, in which case the expression mutual exclusion constraint loss function is expressed as:

[0179]

[0180] Among them, L expLet β represent the loss function for mutual exclusion constraints of facial expressions, O represent the set of mutually exclusive facial expression pairs, and (i, j) be a mutually exclusive facial expression pair in O, wherein the mutually exclusive facial expression pair contains the i-th facial expression base (the facial expression base with index i) and the j-th facial expression base (the facial expression base with index j). i and β j Let represent the expression coefficients of the i-th and j-th expression bases in the first face image, respectively, and let Ⅱ{.} denote the exponential function, ||.|| 2 This represents the square of the L2 norm. It can be understood that for an exponential function, if L2 is true, then L2{.} = 1; if L2 is false, then L2{.} = 0. Corresponding to the expression, if β... i >β j If true, then II{β i >β j} equals 1, if β i >β j If false, then II{β i >β j} equals 0.

[0181] The above formula allows the expression mutual exclusion constraint loss function to retain only the coefficients of the expression bases that are more likely to appear in the first face image among the mutually exclusive expression pairs. Therefore, when adjusting the network parameters of the coefficient regression network based on the expression mutual exclusion constraint loss function, the coefficient regression network learns mutually exclusive expression pairs that should not appear at the same time, thereby promoting the accurate fitting of expression coefficients, that is, improving the accuracy of expression coefficients.

[0182] It is understandable that the unsupervised training loss function, identity constraint loss function, and facial expression mutual exclusion constraint loss function can be calculated simultaneously or sequentially, and the specific calculation order is not currently limited.

[0183] After obtaining the unsupervised training loss function, identity constraint loss function, and expression mutual exclusion loss function corresponding to the first face image of each frame, the loss function of the coefficient regression network is determined.

[0184] At this point, the expression for the loss function of each coefficient regression network is:

[0185]

[0186] Where L represents the loss function of the coefficient regression network, T is the total number of frames of the first acquired face image, and t represents the t-th frame. Let represent the unsupervised training loss function corresponding to the first face image in frame t. Let represent the identity constraint loss function corresponding to the first face image in frame t. Let λ represent the expression mutual exclusion constraint loss function corresponding to the first face image in frame t. idc λ represents the hyperparameter of the identity constraint loss function.exp These represent the hyperparameters of the facial expression mutual exclusion constraint loss function. Setting hyperparameters for the identity constraint loss function and the facial expression mutual exclusion constraint loss function can improve the accuracy of the loss function. The specific values ​​of each hyperparameter can be set according to the actual situation, for example, The importance is higher than Therefore, set λ idc For 1000, λ exp It is 10.

[0187] Step 350: Update the network parameters of the coefficient regression network according to the loss function.

[0188] As described above, mutually exclusive expression pairs were collected using prior knowledge. To address the issue of co-occurrence of expression bases in mutually exclusive expression pairs, a mutual exclusion constraint loss function was designed to inject knowledge of mutually exclusive expressions into the training process of the coefficient regression network. This enabled the coefficient regression network to learn to suppress the co-occurrence of mutually exclusive expressions, thereby further improving the accuracy of expression coefficients.

[0189] The following is an exemplary description of a network training method for face reconstruction. Figure 5 This is a schematic diagram illustrating the framework of a network training method for face reconstruction, provided as an embodiment of this application. (Reference) Figure 5 Multiple frames of first face images are randomly extracted from a video dataset (currently 4 frames), and each frame is input into a corresponding coefficient regression network. These networks share network parameters, and the coefficient regression networks output face reconstruction coefficients, which include an identity coefficient α, where α ∈ R. 80 (There are 80 identity bases), expression coefficient β, α∈R 46 (There are 46 facial bases), texture coefficients δ, δ∈R 80 (With 80 texture bases), illumination coefficient γ, γ∈R 27 (There are 27 lighting types), pose coefficients p, p∈R 6 (Coordinates and angles in three directions). Then, based on the face reconstruction coefficients and the 3D deformed face model (including average face shape and average face texture), a 3D face image (including 3D face shape and 3D face texture) is obtained. Next, a differentiable renderer is used to render the 2D reconstructed image. Finally, an identity constraint loss function is constructed based on the identity coefficients output by the regression network. Figure 5 The middle is denoted as L idc Based on the facial expression coefficients output by the regression network, a mutual exclusion constraint loss function for facial expressions is constructed. Figure 5 The middle is denoted as L expThe network parameters of the coefficient regression network are updated by combining the two-dimensional reconstructed image, the first face image, and the face reconstruction coefficients. The network parameters are then updated by combining the unsupervised training loss function, the identity constraint loss function, and the expression mutual exclusion constraint loss function. After that, training continues until the number of iterations (i.e. the number of times training stops) is satisfied.

[0190] In one embodiment, after training the coefficient regression network, the coefficient regression network can be applied. This involves deploying the coefficient regression network on a suitable device, which could be a network training device for face reconstruction or other devices. The following description uses the application of a coefficient regression network to a network training device for face reconstruction as an example.

[0191] Figure 6 This is a schematic diagram illustrating the application process of a coefficient regression network according to one embodiment of this application. (Reference) Figure 6 When applying coefficient regression networks, the following steps are included:

[0192] Step 410: Obtain the second face image of the second target object.

[0193] For example, the second target object is the object whose virtual portrait needs to be constructed during the application process. The second face image is an image containing the face of the second target object. The source of the second face image is not currently limited. For example, in a video communication scenario, a camera is used to capture the video. In this case, the object displayed in the captured video data (i.e., a human) is the second target object. For the current frame image in the video data, it is first determined whether it contains the face image of the second target object. If it does, the face image is extracted as the second face image; otherwise, the current frame image is not processed. The method of determining whether a face image is contained is not currently limited. For example, a deep learning-based face detection algorithm can be used to determine whether a face region is detected in the current frame image. If a face region is detected, it is determined that a face image is contained; otherwise, it is determined that a face image is not contained. After detecting the face region, the face region can be extracted, that is, the current frame image is cropped to obtain the second face image. For example, a face detector can be used to check whether the current frame image contains a face bounding box. If a face bounding box is found, it is determined that a face image is contained; otherwise, it is determined that a face image is not contained. Then, the face region within the face bounding box is cropped to obtain a second face image.

[0194] Step 420: Input the second face image into the coefficient regression network to obtain the face transfer coefficients of the second face image. The face transfer coefficients include expression coefficients and pose coefficients.

[0195] For example, since the trained coefficient regression networks have the same network parameters, one of the coefficient regression networks can be directly selected during application. Then, the second face image is input to this coefficient regression network, and its output is obtained. In one embodiment, the face reconstruction coefficients output by the coefficient regression network during application are recorded as face transfer coefficients. It is understood that, depending on the actual situation, some face reconstruction coefficients can be set as face transfer coefficients. For example, if expression transfer is required, expression coefficients are selected as face transfer coefficients. In this case, the coefficient regression network can output only expression coefficients, or it can output all face reconstruction coefficients. Then, the device only obtains the expression coefficients as face transfer coefficients. Similarly, if expression and pose transfer are required, expression and pose coefficients are selected as face transfer coefficients. In this case, the coefficient regression network can output only expression and pose coefficients, or it can output all face reconstruction coefficients. Then, the device only obtains the expression and pose coefficients as face transfer coefficients.

[0196] Step 430: Generate a virtual portrait corresponding to the second face image based on the expression coefficient and pose coefficient.

[0197] For example, a virtual image is pre-stored. This virtual image can be a cartoon image, and it can be a two-dimensional or three-dimensional virtual image. In one embodiment, a three-dimensional virtual image is used as an example, and the facial expression bases of the three-dimensional virtual image are also stored. For example, the virtual image is processed according to facial expression coefficients and pose coefficients to generate a virtual portrait with the same facial expression and head movement posture as the second face image. After the virtual portrait is generated, it can be displayed on the device. In one embodiment, the virtual portrait can also be generated by other devices. In this case, the facial expression coefficients and pose coefficients are sent to other devices, and the other devices apply the facial expression coefficients and pose coefficients to the virtual image to generate the virtual portrait.

[0198] In one embodiment, when generating a virtual portrait corresponding to the second face image, a pre-defined 3D virtual image can be rendered and displayed using a graphics rendering framework from an open-source graphics library (Open Graphics Library, OpenGL). Specifically, during rendering, the expression basis of the 3D virtual image is linearly weighted according to expression coefficients to obtain a 3D virtual image containing the expression. After generating the 3D virtual image containing the expression, the graphics rendering framework performs a corresponding rigid transformation on the 3D virtual image containing the expression according to pose parameters to obtain the virtual portrait, which is then displayed.

[0199] For example, Figure 7 This is a diagram of a virtual image generation framework provided in one embodiment of this application. (Reference) Figure 7For a single frame of image, a face detector is used to detect face bounding boxes and the face region is cropped to obtain a second face image containing only the face. Then, the second face image is input into a coefficient regression network to obtain pose coefficients and expression coefficients. Finally, the pose coefficients and expression coefficients are transferred to a virtual image to obtain a virtual portrait.

[0200] Understandably, in video communication scenarios, after generating the virtual image of the current frame, the next frame image can be obtained and used as the current frame image. This process is repeated to continuously display the virtual image.

[0201] As mentioned above, in the application of coefficient regression networks, the expression coefficients and pose coefficients obtained by using coefficient regression networks have high accuracy. As a result, the virtual portrait obtained based on the expression coefficients and pose coefficients is more similar to the expression and head movement posture of the second face image, which effectively protects the privacy of the target object and prevents information leakage. At the same time, the virtual image accurately follows the expression and pose in the second face image.

[0202] Figure 8 This is a schematic diagram of a network training device for face reconstruction provided in one embodiment of this application. (Reference) Figure 8 The network training device for face reconstruction includes: an image acquisition module 501, an image input module 502, an image reconstruction module 503, a loss construction module 504, and a parameter update module 505.

[0203] The system includes: an image acquisition module 501 for acquiring multiple frames of first face images of the first target object; an image input module 502 for inputting each frame of the first face image into the corresponding coefficient regression network to obtain face reconstruction coefficients for each frame of the first face image, wherein each coefficient regression network has the same network parameters and the face reconstruction coefficients include identity coefficients and expression coefficients; an image reconstruction module 503 for obtaining a two-dimensional reconstructed image of the corresponding frame of the first face image based on the face reconstruction coefficients; a loss construction module 504 for constructing a loss function for the coefficient regression network based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients, wherein the loss function includes an unsupervised training loss function for each frame of the first face image and an identity constraint loss function; and a parameter update module 505 for updating the network parameters of the coefficient regression network based on the loss function.

[0204] In one embodiment of this application, the face reconstruction coefficients further include: texture coefficients and pose coefficients. The image reconstruction module 503 includes: a three-dimensional reconstruction unit, used to reconstruct the three-dimensional face shape of the first face image of the corresponding frame according to the identity coefficients and expression coefficients, and to reconstruct the three-dimensional face texture of the first face image of the corresponding frame according to the texture coefficients; and a rendering unit, used to render the two-dimensional reconstructed image of the first face image of the corresponding frame according to the pose coefficients, the three-dimensional face shape and the three-dimensional face texture.

[0205] In one embodiment of this application, the three-dimensional reconstruction unit includes: a shape reconstruction subunit, used to apply the identity coefficient and the expression coefficient to the corresponding identity base and expression base respectively, and combine them with a pre-constructed average face shape to obtain the three-dimensional face shape of the first face image of the corresponding frame; and a texture reconstruction subunit, used to apply the texture coefficient to the corresponding texture base, and combine them with a pre-constructed average face texture to obtain the three-dimensional face texture of the first face image of the corresponding frame.

[0206] In one embodiment of this application, the rendering unit includes: a pose coefficient application subunit, used to apply pose coefficients to a perspective projection model; and a renderer rendering subunit, used to input the perspective projection model, the three-dimensional face shape, and the three-dimensional face texture into the renderer, and to render a two-dimensional reconstructed image of the first face image of the corresponding frame through the renderer.

[0207] In one embodiment of this application, the face reconstruction coefficients further include: illumination coefficients; the device further includes: an illumination coefficient application module, used to apply the illumination coefficients to the illumination model before rendering the two-dimensional reconstructed image of the first face image of the corresponding frame based on the pose coefficients, the three-dimensional face shape and the three-dimensional face texture; and an illumination addition module, used to input the three-dimensional face texture into the illumination model, and add the ambient lighting of the first face image to the three-dimensional face texture through the illumination model.

[0208] In one embodiment of this application, when the loss construction module 504 constructs the identity constraint loss function, it includes: a label coefficient acquisition unit, used to acquire the identity label coefficients corresponding to each frame of the first face image; a coefficient averaging unit, used to use the average coefficient of each identity label coefficient as the pseudo-label supervision coefficient; and a first calculation unit, used to calculate the mean square error between the identity coefficients of the first face image and the pseudo-label supervision coefficients, and obtain the identity constraint loss function based on the mean square error.

[0209] In one embodiment of this application, the unsupervised training loss function includes at least one of the following: an image reconstruction loss function for a first face image, an image perception loss function, a key point loss function, and a regularization loss function.

[0210] The unsupervised training loss function includes: an image reconstruction loss function for the first face image. When constructing the unsupervised training loss function, the loss construction module 504 specifically calculates the pixel error between the first face image and the corresponding two-dimensional reconstructed image using the L2 norm, and obtains the image reconstruction loss function of the first face image based on the pixel error. The unsupervised training loss function also includes: an image perception loss function for the first face image. When constructing the unsupervised training loss function, the loss construction module 504 includes: a feature extraction unit, used to extract first image features of the first face image and second image features of the corresponding two-dimensional reconstructed image using the face recognition network; and a second calculation unit, used to calculate the similarity between the first image features and the second image features, and obtain the image perception loss function of the first face image based on the similarity. The unsupervised training loss function includes: a keypoint loss function for the first face image. The loss construction module 504, when constructing the unsupervised training loss function, includes: a keypoint acquisition unit for acquiring the first face keypoints corresponding to the first face image and the second face keypoints of the corresponding two-dimensional reconstructed image; a third calculation unit for calculating the mean square error of the positions between the first and second face keypoints, and obtaining the keypoint loss function of the first face image based on the mean square error. The unsupervised training loss function also includes: a regularization loss for the first face image. The loss construction module 504, when constructing the unsupervised training loss function, includes: a fourth calculation unit for calculating the squared first norm of the identity coefficient and the squared second norm of the texture coefficient of the first face image, and obtaining the identity and texture constraint loss function of the first face image based on the squared first and second norms; a fifth calculation unit for constraining the expression coefficients of the first face image using the L1 norm to obtain the expression constraint loss function of the first face image; and a sixth calculation unit for obtaining the regularization loss of the first face image based on the identity and texture constraint loss function and the expression constraint loss function.

[0211] In one embodiment of this application, the loss function further includes an expression mutual exclusion constraint loss function for each frame of the first face image. When the loss construction module 504 constructs the expression mutual exclusion constraint loss function, it further includes: a mutual exclusion acquisition unit, used to acquire all pre-constructed mutual exclusion expression pairs, each mutual exclusion expression pair containing at least two mutual exclusion expression bases; a coefficient lookup unit, used to look up the expression coefficients of the expression bases contained in each mutual exclusion expression pair in the first face image; and an eighth calculation unit, used to obtain the expression mutual exclusion constraint loss function of the first face image using an exponential function based on the found expression coefficients.

[0212] In one embodiment of this application, the expression for the expression mutual exclusion constraint loss function is:

[0213]

[0214] Among them, L expLet β represent the loss function for mutual exclusion constraints of facial expressions, O represent the set of mutually exclusive facial expression pairs, (i, j) be a mutually exclusive facial expression pair in O, and the mutually exclusive facial expression pair contains the i-th facial expression base and the j-th facial expression base. i and β j Let represent the expression coefficients of the i-th and j-th expression bases in the first face image, respectively, and let Ⅱ{.} denote the exponential function, ||.|| 2 This represents the square of the L2 norm.

[0215] In one embodiment of this application, after the coefficient regression network is trained, it further includes: a second face acquisition module, used to acquire a second face image of a second target object; a migration coefficient determination module, used to input the second face image into the coefficient regression network to obtain the face migration coefficient of the second face image, the face migration coefficient including: expression coefficient and pose coefficient; and a virtual portrait generation module, used to generate a virtual portrait corresponding to the second face image based on the expression coefficient and pose coefficient.

[0216] The network training device for face reconstruction provided above can be used to execute the network training method for face reconstruction provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0217] It is worth noting that in the above embodiments of the network training device for face reconstruction, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0218] Figure 9 This is a schematic diagram of the structure of a network training device for face reconstruction provided in an embodiment of this application. Figure 9 As shown, the network training device for face reconstruction includes a processor 50, a memory 51, an input device 52, and an output device 53; the number of processors 50 in the network training device for face reconstruction can be one or more. Figure 9 Taking a processor 50 as an example, in a network training device for face reconstruction, the processor 50, memory 51, input device 52, and output device 53 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0219] The memory 51, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the network training method for face reconstruction in the embodiments of this application (e.g., the image acquisition module 501, image input module 502, image reconstruction module 503, loss construction module 504, and parameter update module 505 in the network training device for face reconstruction). The processor 50 executes various functional applications and data processing of the network training device for face reconstruction by running the software programs, instructions, and modules stored in the memory 51, thereby implementing the aforementioned network training method for face reconstruction.

[0220] The memory 51 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the network training device for face reconstruction. Furthermore, the memory 51 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 51 may further include memory remotely located relative to the processor 50, which can be connected to the network training device for face reconstruction via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0221] Input device 52 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the network training device for face reconstruction. It may also include an image acquisition device, an audio acquisition device, etc. Output device 53 may include a display screen or other display device, and may also include an audio playback device. In addition, the network training device for face reconstruction may also include a communication device to realize data communication with remote devices, such as video communication.

[0222] The aforementioned network training device for face reconstruction includes a network training apparatus for face reconstruction, which can be used to execute any network training method for face reconstruction and has corresponding functions and beneficial effects.

[0223] Furthermore, embodiments of this application also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the network training method for face reconstruction provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0224] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0225] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0226] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0227] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0228] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0229] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A network training method for face reconstruction, characterized in that, include: Acquire multiple frames of the first face image of the first target object; Each frame of the first face image is input into the corresponding coefficient regression network to obtain the face reconstruction coefficients of each frame of the first face image. Each coefficient regression network has the same network parameters. The face reconstruction coefficients include identity coefficients and expression coefficients. The two-dimensional reconstructed image of the first face image in the corresponding frame is obtained based on the face reconstruction coefficients; The loss function of the coefficient regression network is constructed based on the first face image, the corresponding 2D reconstructed image, and the face reconstruction coefficients. The loss function includes an unsupervised training loss function and an identity constraint loss function for each frame of the first face image. The first face image has identity label coefficients. The identity constraint loss function is constructed based on the identity label coefficients and the identity coefficients output by the coefficient regression network. The loss function also includes an expression mutual exclusion constraint loss function for each frame of the first face image. The expression mutual exclusion constraint loss function is constructed using an exponential function, and its expression is: in, L exp This represents the loss function for mutual exclusion constraints of facial expressions. O Let (i, j) be the set of mutually exclusive expression pairs. O A mutually exclusive emoji pair, wherein the mutually exclusive emoji pair contains an i-th emoji base and a j-th emoji base. β i and β j Let |i| represent the expression coefficients of the i-th and j-th expression bases in the first face image, respectively, and let |i| denote an exponential function. 2 Represents the square of the L2 norm; The network parameters of the coefficient regression network are updated according to the loss function.

2. The method according to claim 1, characterized in that, The face reconstruction coefficients also include texture coefficients and pose coefficients. The step of obtaining the two-dimensional reconstructed image of the first face image corresponding to the frame based on the face reconstruction coefficients includes: The three-dimensional face shape of the first face image in the corresponding frame is reconstructed based on the identity coefficient and the expression coefficient, and the three-dimensional face texture of the first face image in the corresponding frame is reconstructed based on the texture coefficient. The two-dimensional reconstructed image of the first face image in the corresponding frame is obtained by rendering the pose coefficient, the three-dimensional face shape, and the three-dimensional face texture.

3. The method according to claim 2, characterized in that, The step of reconstructing the three-dimensional face shape of the corresponding frame of the first face image based on the identity coefficient and the expression coefficient includes: The identity coefficient and the expression coefficient are applied to the corresponding identity base and expression base respectively, and combined with the pre-constructed average face shape to obtain the three-dimensional face shape of the first face image of the corresponding frame; The step of reconstructing the three-dimensional face texture of the corresponding frame of the first face image based on the texture coefficients includes: The texture coefficients are applied to the corresponding texture base and combined with the pre-constructed average face texture to obtain the three-dimensional face texture of the first face image of the corresponding frame.

4. The method according to claim 2, characterized in that, The step of rendering the two-dimensional reconstructed image of the first face image corresponding to the frame based on the pose coefficient, the three-dimensional face shape, and the three-dimensional face texture includes: The pose coefficients are applied to the perspective projection model; The perspective projection model, the three-dimensional face shape, and the three-dimensional face texture are input into the renderer, and the renderer renders the two-dimensional reconstructed image of the first face image in the corresponding frame.

5. The method according to claim 2 or 4, characterized in that, The face reconstruction coefficients also include illumination coefficients; Before rendering the two-dimensional reconstructed image of the first face image of the corresponding frame based on the pose coefficients, the three-dimensional face shape, and the three-dimensional face texture, the method further includes: Apply the illumination coefficients to the illumination model; The three-dimensional face texture is input into the lighting model, and the ambient lighting of the first face image is added to the three-dimensional face texture through the lighting model.

6. The method according to claim 1, characterized in that, The identity constraint loss function of the coefficient regression network constructed based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients includes: Obtain the identity label coefficients corresponding to the first face image in each frame; The average coefficient of each of the aforementioned identity label coefficients is used as the pseudo-label supervision coefficient; Calculate the mean square error between the identity coefficient of the first face image and the pseudo-label supervision coefficient, and obtain the identity constraint loss function based on the mean square error.

7. The method according to claim 2, characterized in that, The unsupervised training loss function includes at least one of the following: an image reconstruction loss function for the first face image, an image perception loss function, a key point loss function, and a regularization loss function. The unsupervised training loss function includes: the image reconstruction loss function of the first face image. The unsupervised training loss function is constructed based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients, including: The pixel error between the first face image and the corresponding two-dimensional reconstructed image is calculated using the L2 norm, and the image reconstruction loss function of the first face image is obtained based on the pixel error. The unsupervised training loss function includes: the image perception loss function of the first face image. The unsupervised training loss function is constructed based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients, including: The first image features of the first face image and the second image features of the corresponding two-dimensional reconstructed image are extracted using a face recognition network. Calculate the similarity between the first image feature and the second image feature, and obtain the image perception loss function of the first face image based on the similarity; The unsupervised training loss function includes: the key point loss function of the first face image. The unsupervised training loss function is constructed based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients, including: Obtain the first facial key points corresponding to the first facial image and the second facial key points of the corresponding two-dimensional reconstructed image; Calculate the mean square error of the positions between the first facial key points and the second facial key points, and obtain the key point loss function of the first facial image based on the mean square error; The unsupervised training loss function includes: a regularization loss function for the first face image. The unsupervised training loss function is constructed based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients, including: Calculate the squared first norm of the identity coefficient and the squared second norm of the texture coefficient of the first face image, and obtain the identity and texture constraint loss function of the first face image based on the squared first norm and the squared second norm. The L1 norm is used to constrain the expression coefficients of the first face image to obtain the expression constraint loss function of the first face image. The regularization loss function of the first face image is obtained based on the identity and texture constraint loss function and the expression constraint loss function.

8. The method according to claim 1, characterized in that, The expression mutual exclusion constraint loss function is constructed based on the first face image, the corresponding two-dimensional reconstructed image, and the face reconstruction coefficients, including: Retrieve all pre-built mutually exclusive emoji pairs; Find the expression coefficients of the expression base contained in each of the mutually exclusive expression pairs in the first face image; Based on the found expression coefficients, the expression mutual exclusion constraint loss function of the first face image is obtained using an exponential function.

9. The method according to claim 2, characterized in that, After the coefficient regression network is trained, it also includes: Obtain the second face image of the second target object; The second face image is input into the coefficient regression network to obtain the face transfer coefficient of the second face image, which includes expression coefficient and pose coefficient; A virtual portrait corresponding to the second face image is generated based on the expression coefficient and the pose coefficient.

10. A network training device for face reconstruction, characterized in that, include: The image acquisition module is used to acquire multiple frames of the first face image of the first target object; An image input module is used to input each frame of the first face image into a corresponding coefficient regression network to obtain the face reconstruction coefficients of each frame of the first face image. Each coefficient regression network has the same network parameters, and the face reconstruction coefficients include identity coefficients and expression coefficients. The image reconstruction module is used to obtain a two-dimensional reconstructed image of the first face image in the corresponding frame based on the face reconstruction coefficients. The loss construction module is used to construct a loss function for the coefficient regression network based on the first face image, the corresponding 2D reconstructed image, and the face reconstruction coefficients. The loss function includes an unsupervised training loss function and an identity constraint loss function for each frame of the first face image. The first face image has identity label coefficients. The identity constraint loss function is constructed based on the identity label coefficients and the identity coefficients output by the coefficient regression network. The loss function also includes an expression mutual exclusion constraint loss function for each frame of the first face image. The expression mutual exclusion constraint loss function is constructed using an exponential function, and its expression is: in, L exp This represents the loss function for mutual exclusion constraints of facial expressions. O Let (i, j) be the set of mutually exclusive expression pairs. O A mutually exclusive emoji pair, wherein the mutually exclusive emoji pair contains an i-th emoji base and a j-th emoji base. β i and β j Let |i| represent the expression coefficients of the i-th and j-th expression bases in the first face image, respectively, and let |i| denote an exponential function. 2 Represents the square of the L2 norm; The parameter update module is used to update the network parameters of the coefficient regression network according to the loss function.

11. A network training device for face reconstruction, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the network training method for face reconstruction as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the network training method for face reconstruction as described in any one of claims 1-9.

Images