Method and application of 3D face reconstruction model based on weakly supervised learning

Abstract

The invention discloses a method for establishing a three-dimensional face reconstruction model based on weakly supervised learning, which belongs to the technical field of three-dimensional face reconstruction, and includes: establishing a weakly supervised learning model for three-dimensional face reconstruction, including: a three-dimensional face reconstruction network, which is used to extract three-dimensional face parameters from an input image; a face feature extraction model, which is used to extract a face feature vector from an input image; a GCN optimized decoder, which is used to optimize the coarse texture parameters according to the face feature vector to obtain fine texture parameters; and a three-dimensional face generator, which is used to generate a three-dimensional face model according to shape parameters and fine texture parameters; obtain a face image data set, and train the model; during the training process, the three-dimensional face model output by the model is rendered as a two-dimensional face image and a loss value is calculated, and the model parameters are optimized to minimize the error between the two-dimensional face image and the input image. The invention can improve the texture clarity of the three-dimensional face model and can be applied to traditional Chinese medicine face diagnosis.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional face reconstruction, and more specifically, relates to a method for establishing a three-dimensional face reconstruction model based on weakly supervised learning and an application thereof. Background Art

[0002] TCM diagnosis is a set of systematic medical theories summarized by TCM experts based on thousands of years of clinical experience. The four methods of "looking, smelling, asking and feeling" are the main ways for TCM to understand and study diseases. As an important part of facial diagnosis, facial diagnosis has always been the focus of research in the field of traditional TCM diagnosis. The face is both the glory of the internal organs, qi and blood, and the place where the meridians converge. The morphology, gloss, texture and other features of the face can well reflect the running status of the human body's qi and blood, and characterize the health status of the internal organs of the human body. However, this traditional facial diagnosis method, on the one hand, is heavily dependent on the professional knowledge and clinical experience accumulated by TCM experts over a long period of time, and lacks objective and quantitative evaluation indicators; on the other hand, it is limited by the external environment such as lighting and noise during the doctor's face diagnosis, which is not conducive to follow-up reference and experience sharing, and seriously hinders the popularization and promotion of TCM diagnosis.

[0003] As computer and artificial intelligence technologies are widely used in many fields such as medical image processing and medical information systems and play an important role, using computer technology to process and analyze facial images to achieve automation, objectivity and standardization of the TCM facial diagnosis process is an important research direction and modern development trend of TCM facial diagnosis. Compared with two-dimensional facial images, three-dimensional facial images carry richer personalized features and have important research significance and application prospects.

[0004] In the study of TCM facial images, the reconstruction of 3D facial images can achieve more personalized feature extraction and more accurate facial region positioning to analyze the relationship between facial features and corresponding internal organs. However, the existing 3D facial reconstruction methods focus on reconstructing the 3D shape of the face, but the reconstructed 3D facial models often have unclear textures and are not suitable for TCM facial diagnosis. Summary of the invention

[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a method for establishing a three-dimensional face reconstruction model based on weakly supervised learning and its application, which aims to effectively solve the technical problem that the three-dimensional face model reconstructed by the existing three-dimensional face reconstruction method has unclear texture and is not suitable for applications such as traditional Chinese medicine facial diagnosis that rely on clear facial texture information.

[0006] To achieve the above object, according to one aspect of the present invention, a method for establishing a 3D face reconstruction model based on weakly supervised learning is provided, comprising:

[0007] A weakly supervised learning model for three-dimensional face reconstruction is established, the weakly supervised learning model comprising: a three-dimensional face reconstruction network, used to extract three-dimensional face parameters from an input image, the input image contains a face image, and the three-dimensional face parameters include shape parameters and coarse texture parameters; a face feature extraction model, used to extract a face feature vector from the input image; a GCN optimized decoder, used to optimize the coarse texture parameters according to the face feature vector to obtain fine texture parameters; and a three-dimensional face generator, used to generate a three-dimensional face model according to the shape parameters and the fine texture parameters;

[0008] A face image dataset is obtained and a weakly supervised learning model is trained. During the training process, the three-dimensional face model output by the weakly supervised learning model is rendered into a two-dimensional face image, and a loss value is calculated based on the error between the rendered two-dimensional face image and the input image. The model parameters of the model are optimized according to the calculated loss value to minimize the error between the two-dimensional face image and the input image. After the training, a three-dimensional face reconstruction model is obtained.

[0009] Furthermore, during the training process, the loss function L used is:

[0010] L=ε1[L p +ε2L i ]+ε3L v

[0011] Among them, L p , L i and L v They respectively represent the color loss, perceptual loss and vertex loss of the rendered two-dimensional face image relative to the input image. ε1, ε2 and ε3 are weight constants used to control several losses.

[0012] Furthermore, the three-dimensional face reconstruction network is a pre-trained 3DMM regressor, and the coarse texture parameters include texture parameters, expression parameters, illumination parameters and posture parameters.

[0013] Furthermore, the face feature extraction model includes a face region segmentation module and a feature extraction module;

[0014] The face region segmentation module is used to segment the face region from the input image;

[0015] The feature extraction module is used to extract facial feature vectors from the face area.

[0016] Furthermore, the feature extraction module is a model obtained by removing the structure of the Flatten layer from the pre-trained FaceNet model.

[0017] Furthermore, during the training process, a differentiable renderer is used to render the three-dimensional face model output by the weakly supervised learning model into a two-dimensional face image.

[0018] Furthermore, the differentiable renderer is SoftRas.

[0019] According to another aspect of the present invention, a three-dimensional face reconstruction method is provided, comprising:

[0020] An image containing a human face is input into the three-dimensional human face reconstruction model established by the three-dimensional human face reconstruction model establishment method based on weakly supervised learning provided by the present invention to reconstruct the corresponding three-dimensional human face model.

[0021] Furthermore, the three-dimensional face reconstruction method provided by the present invention is used for traditional Chinese medicine facial diagnosis.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, comprising a stored computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for establishing a three-dimensional face reconstruction model based on weakly supervised learning provided by the present invention, and / or the above-mentioned three-dimensional face reconstruction method provided by the present invention.

[0023] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0024] (1) After extracting the three-dimensional face parameters from the face image, the present invention further extracts the face feature vector from the face image, and optimizes the coarse texture parameters in the three-dimensional face parameters based on the extracted face feature vector to obtain more refined fine texture parameters, and finally generates a three-dimensional face model based on the shape parameters in the three-dimensional face parameters and the optimized fine texture parameters. Since the texture parameters are optimized and refined when generating the three-dimensional face model, the texture clarity of the three-dimensional face model reconstructed by the present invention is effectively improved.

[0025] (2) In the process of training the model for three-dimensional face reconstruction, the present invention renders the three-dimensional face model output by the model into a two-dimensional face image, and calculates the loss value based on the error between the two-dimensional face image and the input image. The input image is directly used as the basis for loss calculation. On the one hand, it can ensure that the reconstruction accuracy of the model is high. On the other hand, it realizes a weakly supervised training method without the need for pre-labeling, thereby reducing the requirements for training samples.

[0026] (3) When the present invention performs model training, the calculated loss function includes the color loss, perceptual loss and vertex loss of the two-dimensional face image relative to the input image. These three types of losses respectively reflect the color difference of the pixels between the rendered image and the input image, the deep feature difference between the rendered image and the input image, and the vertex error in the reconstructed three-dimensional face model. Therefore, the present invention performs model training based on the loss function, which can effectively utilize the pixel information of the two-dimensional face image and realize weak supervision at the pixel level. The reconstruction accuracy of the trained model is relatively high.

[0027] (4) The present invention first segments the face region from the input image, and then extracts the face feature vector from the face region. Through image segmentation, the interference of redundant information can be reduced and the expression ability of the face feature vector can be improved.

[0028] (5) The present invention uses a 3DMM regressor as a three-dimensional face reconstruction network to extract three-dimensional face parameters from an input image. The extracted three-dimensional face parameters specifically include shape parameters, texture parameters, expression parameters, illumination parameters and posture parameters. The present invention uses other parameters other than shape parameters, namely, texture parameters, expression parameters, illumination parameters and posture parameters, as coarse texture parameters and optimizes them using a GCN decoding optimizer. The obtained fine texture parameters can reflect the influence of external factors such as illumination on texture information, thereby further improving the accuracy and clarity of texture information in the reconstructed three-dimensional face model.

[0029] (6) The present invention can be applied to fields such as traditional Chinese medicine facial diagnosis that have high requirements for texture clarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a method for establishing a 3D face reconstruction model based on weakly supervised learning provided by an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the existing GCN decoding optimizer network structure;

[0032] Figure 3 This is a schematic diagram of the structure of an existing differentiable renderer;

[0033] Figure 4 A schematic diagram of the three-dimensional face reconstruction effect provided in an embodiment of the present invention; wherein (a) is an input face image, (b) to (d) are views of the three-dimensional face model reconstructed by the model at different angles, and (e) is a two-dimensional face image obtained by rendering the three-dimensional face model reconstructed by the model. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0035] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0036] In order to solve the technical problem that the texture of the three-dimensional face model reconstructed by the existing three-dimensional face reconstruction method is unclear and is not suitable for applications such as traditional Chinese medicine facial diagnosis that rely on clear facial texture information, the present invention provides a method for establishing a three-dimensional face reconstruction model based on weak supervised learning and its application. The overall idea is to optimize the texture information in the three-dimensional face parameters and generate a three-dimensional face model using the optimized three-dimensional face parameters to improve the clarity of the texture information in the three-dimensional face model.

[0037] The following are examples.

[0038] Embodiment 1:

[0039] A method for establishing a three-dimensional face reconstruction model based on weakly supervised learning includes: a model establishment step and a model training step.

[0040] In this embodiment, the model building step specifically includes:

[0041] Establish a weakly supervised learning model for 3D face reconstruction. The weakly supervised learning model is as follows: Figure 1 As shown, it includes: a 3D face reconstruction network, a face feature extraction model, a GCN optimized decoder and a 3D face generator, wherein:

[0042] A 3D face reconstruction network is used to extract 3D face parameters from an input image, the input image contains a face image, and the 3D face parameters include shape parameters and coarse texture parameters; as an optional implementation, Figure 1 As shown, in this embodiment, the 3D face reconstruction model is a pre-trained 3DMM regressor, and the extracted 3D face parameters include shape parameters, texture parameters, expression parameters, illumination parameters and posture parameters. Optionally, in this embodiment, the 3D face parameters are expressed as:

[0043]

[0044] represents the shape parameter, Indicates expression parameters, Represents texture parameters, Represents the lighting parameters, Represents posture parameters; 3DMM uses the PCA (Principal Component Analysis) method to perform linear combinations of shape vectors and texture vectors; the calculation formulas for the shape S and texture T of the 3D face to be reconstructed are:

[0045]

[0046]

[0047] In the formula, and They are the average face shape and texture in the BFM (Basel Face Model) database, B i , B e and B t are the PCA basis vectors of the shape, expression and texture of the three-dimensional face after standard deviation scaling; the traditional three-dimensional face reconstruction method often directly generates a corresponding three-dimensional face model based on these parameters after extracting the above three-dimensional face parameters, and the texture of the three-dimensional face model reconstructed in this way is often unclear; in this embodiment, other parameters other than the shape parameters, namely, texture parameters, expression parameters, illumination parameters and posture parameters are unified as coarse texture parameters, and the coarse texture parameters are further optimized by using the remaining structures in the model; it should be noted that the 3DMM regressor is only an optional implementation of the present invention and should not be understood as the only limitation of the present invention. Other methods that can extract three-dimensional face parameters from face images can also be used in the present invention;

[0048] The face feature extraction model is used to extract the face feature vector from the input image; Figure 1 As shown, in this embodiment, the face feature extraction model specifically includes a pre-trained face region segmentation module and a feature extraction module. The face region segmentation module is used to segment the face region from the input image to reduce the interference of redundant information. In this embodiment, the face region segmentation module can be implemented by any image segmentation model.

[0049] The feature extraction module is used to extract a facial feature vector from a facial region. Optionally, in this embodiment, the network structure after the Flatten layer in the pre-trained FaceNet model is removed, and the remaining structure is used as a feature extraction module. In the embodiment, the FaceNet pre-trained model is an official model trained using the CASIA-WebFace data set. It should be noted that the feature extraction module constructed here is only an optional implementation method and should not be understood as the only limitation of the present invention. Any other model that can extract a facial feature vector from a facial region can be used in the present invention.

[0050] The GCN optimized decoder is used to optimize the coarse texture parameters according to the face feature vector to obtain the fine texture parameters; the structure of the GCN optimized decoder is as follows Figure 2 As shown in the figure, it includes a GCN decoder and a GCN optimizer. The GCN decoder uses four spectral residual blocks, each of which contains two Chebyshev convolutional layers and uses ReLU activation; the GCN optimizer uses a similar GCN spectral residual block, and adds an upsampling layer and a downsampling layer at the top and bottom respectively; the face feature vector is used as the input of the GCN decoder, and the coarse texture parameters are used as the input of the GCN optimizer. The outputs of the GCN decoder and the GCN optimizer are connected in series and input into the Chebyshev convolutional layer to optimize the coarse texture parameters and output the fine texture parameters;

[0051] A 3D face generator is used to generate a 3D face model according to shape parameters and fine texture parameters; since the texture parameters are optimized, the 3D face model generated by the 3D face generator is also an optimized 3D face model, in which the texture is clearer; as an optional implementation, this embodiment directly uses a pre-trained 3DMM regressor as a 3D face generator, and inputs the optimized 3D face parameters composed of shape parameters and fine texture parameters into it to generate a corresponding 3D face model;

[0052] In this embodiment, the model training step specifically includes:

[0053] A face image dataset is obtained to train a weakly supervised learning model; it is easy to understand that in the face image dataset obtained here, the size of the face image should meet the input requirements of the model. Optionally, in this embodiment, the size of the face image in the face image dataset is specifically 600×900. Since the size of the image actually collected by the camera is generally larger, such as 4000×6000, the image needs to be resampled to a specified size; in some other embodiments of the present invention, the size of the model input image can also be set to other sizes as needed;

[0054] In order to quantitatively calculate the reconstruction loss of the model to measure the training effect of the model, such as Figure 1 As shown, in this embodiment, during the training process, the 3D face model output by the weakly supervised learning model is rendered into a 2D face image, and a loss value is calculated based on the error between the rendered 2D face image and the input image, and the model parameters of the model are optimized according to the calculated loss value to minimize the error between the 2D face image and the input image; after the training is completed, a 3D face reconstruction model is obtained;

[0055] Optionally, in this embodiment, the three-dimensional face model output by the weakly supervised learning model is rendered into a two-dimensional face image by SoftRas, which is a differentiable renderer with a structure as shown below: Figure 3 As shown in Figure 1, the process of differentiable rendering of the 3D face model is as follows:

[0056] The input of the feature extraction part is the shape parameter S and texture parameter T of the optimized 3D face model. At the same time, the camera parameter P and the lighting parameter L participate in the regulation of the rendering process as external environment variables. The shape parameter S generates the normal vector N, depth Z and corresponding screen coordinate U of the face mesh through coordinate transformation. In a specific lighting environment L, the vertex color C of the model can be calculated through the mesh normal vector N and the model texture parameter T. The input of the SoftRas rendering process is the mesh depth Z, screen coordinate U and vertex color C. The process first performs probability mapping on the screen coordinate U, calculates the pixel probability D of each point, and then uses the aggregation function to fuse the features of the pixel probability D, mesh depth Z and vertex color C to generate the rendered image I.

[0057] It should be noted that SoftRas is only an optional implementation of the present invention and should not be understood as the only limitation of the present invention. In other embodiments of the present invention, other display rendering methods may also be used.

[0058] In order to further ensure the reconstruction effect of the model, this embodiment designs a hybrid loss function, which is represented by L and is expressed as follows:

[0059] L=ε1[L p +ε2L i ]+ε3L v

[0060] Among them, L p , L i and L vThey respectively represent the color loss, perceptual loss and vertex loss of the rendered two-dimensional face image relative to the input image. These three types of losses respectively reflect the color difference of pixels between the rendered image and the input image, the deep feature difference between the rendered image and the input image, and the vertex error in the reconstructed three-dimensional face model. This embodiment performs model training based on the loss function, which can effectively utilize the pixel information of the two-dimensional face image and realize weak supervision at the pixel level. The reconstruction accuracy of the trained model is relatively high; ε1, ε2 and ε3 are weight constants for controlling several losses. Optionally, in this embodiment, ε2 is set to 0.2, and ε1 and ε3 are gradually adjusted from 0 and 1 to 1 and 0 respectively;

[0061] Specifically, in the mixed loss function L, the three types of losses are calculated as follows:

[0062] In order to compare the difference between the rendered image and the input image, the face area is pre-segmented from the input image in the network structure to reduce the interference of redundant information, and then the pixel loss between the rendered image and the face segmentation image is calculated using the following loss function:

[0063]

[0064] Among them, M represents the face area in the projected image, i represents the pixel position, and A i Indicates the confidence of the i pixel position, provided by Mask, I i and I′ i Respectively represent the colors of the pixels of the input image and the projected image;

[0065] The pre-trained FaceNet network is used to extract the perceptual layer information of the face image, and the semantic level features of the face model are fitted through the following perceptual loss function:

[0066]

[0067] Among them, f(I) and f′(I) represent the deep features of the input 2D face image and the rendered image, respectively, and <, > represent the inner product;

[0068] The vertex loss is calculated by the difference between the vertex color of the rendered image of the 3D reconstructed face model and the input 2D face image. The loss function formula is as follows:

[0069]

[0070]

[0071] Among them, |x| represents the error between the predicted key point and the key point label, ω and ε are parameters of the logarithmic function, ω represents the range of nonlinearity, ε represents the curvature, and C is a constant controlled by ω and ε, which is used to smooth the linear and nonlinear parts of the loss function; optionally, in this embodiment, ω and ε are empirically set to 10 and 2.

[0072] Embodiment 2:

[0073] A three-dimensional face reconstruction method, comprising:

[0074] An image containing a human face is input into the three-dimensional face reconstruction model established by the three-dimensional face reconstruction model establishment method based on weakly supervised learning provided by Example 1 of the present invention to reconstruct a corresponding three-dimensional face model.

[0075] This embodiment can be applied to applications such as traditional Chinese medicine facial diagnosis that have high requirements for texture information.

[0076] Embodiment 3:

[0077] A computer-readable storage medium includes a stored computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for establishing a three-dimensional face reconstruction model based on weakly supervised learning provided in Example 1 of the present invention, and / or the above-mentioned three-dimensional face reconstruction method provided in Example 2 of the present invention.

[0078] The three-dimensional face reconstruction effect of the present invention is further described below in combination with some specific reconstruction results.

[0079] Respectively Figure 4 The multiple face images shown in (a) are used as the input of the 3D face reconstruction model, and the model outputs the corresponding 3D face model. The views of the 3D face model at different angles are shown as follows: Figure 4 As shown in (b), (c) and (d) in Figure 4 It can be seen from the results shown in (b) to (d) that the three-dimensional face model reconstructed by the method of the present invention has good reconstruction effect at all angles, the three-dimensional structural features of the face are obvious, the face texture is relatively clear, and it can effectively reflect the real skin color and texture, and can be applied to traditional Chinese medicine facial diagnosis.

[0080] Each 3D face model is further rendered to obtain the corresponding 2D face image, such as Figure 4 As shown in (e), compared with Figure 4 (e) and Figure 4 As can be seen from (a) in the figure, the reconstructed rendering image has a high correlation with the input image, the corresponding features of the face are well reconstructed, and the detail information is restored significantly, which further illustrates the effectiveness of the method provided by the present invention.

[0081] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for establishing a 3D face reconstruction model based on weakly supervised learning, characterized in that: include: A weakly supervised learning model for three-dimensional face reconstruction is established, the weakly supervised learning model comprising: a three-dimensional face reconstruction network, used to extract three-dimensional face parameters from an input image, the input image contains a face image, the three-dimensional face parameters include shape parameters and coarse texture parameters; a face feature extraction model, used to extract a face feature vector from the input image; a GCN optimized decoder, used to optimize the coarse texture parameters according to the face feature vector to obtain fine texture parameters; and a three-dimensional face generator, used to generate a three-dimensional face model according to the shape parameters and the fine texture parameters; the three-dimensional face reconstruction network is a pre-trained 3DMM regressor, and the coarse texture parameters include texture parameters, expression parameters, illumination parameters and posture parameters; Acquire a face image dataset and train the weakly supervised learning model; during the training process, render the three-dimensional face model output by the weakly supervised learning model into a two-dimensional face image, and calculate a loss value based on the error between the rendered two-dimensional face image and the input image, and optimize the model parameters of the model according to the calculated loss value to minimize the error between the two-dimensional face image and the input image; after the training is completed, obtain a three-dimensional face reconstruction model; During the training process, the loss function L used is: L=ε1[L p +ε2L i ]+ε3L v Among them, L p , L i and L v They represent the color loss, perceptual loss and vertex loss of the rendered two-dimensional face image relative to the input image, respectively. ε1, ε2 and ε3 are weight constants used to control several losses. The loss function formula of vertex loss is as follows: Among them, |x| represents the error between the predicted key point and the key point label, ω and ε are the parameters of the logarithmic function, ω represents the range of nonlinearity, ε represents the curvature, and C is a constant controlled by ω and ε.

2. The method for establishing a 3D face reconstruction model based on weakly supervised learning according to claim 1, characterized in that: The face feature extraction model includes a face region segmentation module and a feature extraction module; The face region segmentation module is used to segment the face region from the input image; The feature extraction module is used to extract a facial feature vector from the facial area.

3. The method for establishing a 3D face reconstruction model based on weakly supervised learning as claimed in claim 2, characterized in that: The feature extraction module is a model obtained by removing the structure of the Flatten layer from the pre-trained FaceNet model.

4. The method for establishing a 3D face reconstruction model based on weakly supervised learning according to claim 1, characterized in that: During the training process, a differentiable renderer is used to render the three-dimensional face model output by the weakly supervised learning model into a two-dimensional face image.

5. The method for establishing a 3D face reconstruction model based on weakly supervised learning as claimed in claim 4, characterized in that: The differentiable renderer is SoftRas.

6. A three-dimensional face reconstruction method, characterized in that: include: An image containing a human face is input into a three-dimensional face reconstruction model established by the method for establishing a three-dimensional face reconstruction model based on weakly supervised learning as described in any one of claims 1 to 5 to reconstruct a corresponding three-dimensional face model.

7. The three-dimensional face reconstruction method according to claim 6, characterized in that: Used for Chinese medicine consultation.

8. A computer-readable storage medium, characterized in that: It includes a stored computer program, which, when executed by a processor, controls the device where the computer-readable storage medium is located to execute the method for establishing a three-dimensional face reconstruction model based on weakly supervised learning as described in any one of claims 1 to 5, and / or the three-dimensional face reconstruction method as described in claim 6 or 7.

Images