A single-image 3D face reconstruction method based on differentiable renderer

Abstract

The present invention discloses a single-image 3D face reconstruction method based on a differentiable renderer, comprising the following steps: S1, inputting a target face image into a pre-trained regression network to obtain initialization 3D face parameters; S2, inputting 3D face parameters and posture parameters into a differentiable renderer to obtain a rendered image with the same posture as the input image and a rendered image with different postures, respectively; S3, comparing the target image with the rendered image under the same posture to obtain a key point loss value and a pixel-level loss value; S4, comparing the target image with the rendered image under different postures to obtain a generative adversarial loss value and an identity consistency loss value; S5, optimizing the 3D face parameters according to the calculated loss value, and returning the updated 3D face parameters to step S2 until the iteration converges to obtain the optimized 3D face parameters. The present invention introduces a truly differentiable renderer to obtain a higher quality 3D face reconstruction result.

Description

Technical Field

[0001] The invention belongs to the field of image processing and relates to a single-image three-dimensional face reconstruction method based on a differentiable renderer. Background Art

[0002] With the continuous development of face recognition technology, 3D face reconstruction technology has gradually developed into an important application branch of computer graphics. Traditional 3D face reconstruction technology mainly relies on expensive 3D scanning equipment and a lot of post-processing. Therefore, how to use 2D face images to quickly and accurately reconstruct 3D face models has become a research focus.

[0003] The most advanced 3D face reconstruction methods can be roughly divided into two categories: learning-based methods and optimization-based methods. Deep learning-based methods usually adopt regression methods, taking face images as input and learning to regress the corresponding 3D face model parameters. However, these methods usually require a large amount of labeled data, and it is difficult to obtain the real 3D face model parameters. On the other hand, optimization-based methods generally regard face imaging as a generation process, taking a series of parameters (including face geometry, albedo, texture, illumination, viewing angle, etc.) as input, generating a rendered image according to certain graphic rules, and optimizing the input parameters by minimizing the distance between the rendered image and the target image.

[0004] Recent developments in differentiable renderers provide an effective tool optimization framework for both types of face reconstruction methods. The regression parameters in learning-based methods can be rendered into images and these parameters can be optimized using pixel loss, which allows for unsupervised training. For optimization-based methods, differentiable renderers introduce gradient-based optimization, allowing for more complex loss functions and stabilizing the training process. However, most existing methods simply use z-buffer rendering, which is not necessarily differentiable, especially when the triangles surrounding each pixel change during the optimization process. These methods result in certain defects in the authenticity and some texture details (hair, eyebrows, wrinkles) of the reconstructed 3D faces. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a single-image 3D face reconstruction method based on a differentiable renderer to solve the problem that 3D face reconstruction cannot be accurately achieved.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A single-image 3D face reconstruction method based on a differentiable renderer comprises the following steps:

[0008] S1. Input the target face image into the pre-trained 3D face model to obtain the initialized 3D face parameters;

[0009] S2, inputting the initialized 3D face parameters and posture parameters into a differentiable renderer, and obtaining a rendered image with the same posture as the input image and a rendered image with a different posture;

[0010] S3, comparing the target face image with the rendered image in the same posture to obtain key point loss values ​​and pixel-level loss values;

[0011] S4, comparing the target face image with the rendered images in different postures to obtain the generated adversarial loss value and the identity consistency loss value;

[0012] S5. Optimize the three-dimensional face parameters according to the calculated loss value, and return the updated three-dimensional face parameters to step S2 until the iteration converges to obtain the optimized three-dimensional face parameters.

[0013] Furthermore, step S1 is specifically as follows:

[0014] Input the target face image into the pre-trained Large Scale Face Model 3D face model, and regress to obtain the initialized 3D face parameters, including identity parameter α, expression parameter β, texture parameter T, and posture parameter p0;

[0015] Among them, the expression of the three-dimensional face model is as follows:

[0016]

[0017] Where S represents the three-dimensional face shape, S represents the average face shape, and s α and β are the face identity and facial expression vector basis respectively, α is the 158-dimensional face identity coefficient, and β is the 29-dimensional facial expression coefficient.

[0018] Furthermore, the attitude parameters are obtained by random sampling, where the pitch and yaw angles are sampled from U(-40°, 40°), and the roll angle is sampled from U(-15°, 15°).

[0019] Furthermore, the initialized three-dimensional face parameters and posture parameters obtained by regression are input into a differentiable renderer to obtain a rendered image G1 having the same posture as the input image and a rendered image G2 having a different posture from the input image.

[0020] Furthermore, the differentiable renderer considers the aggregation mechanism of all triangle probabilistic contributions to handle occlusion. The implementation steps are as follows:

[0021] Construct the probability map to estimate the triangle fj Probability contribution to pixel Pi

[0022]

[0023] in, σ is a scalar that controls the sharpness of the probability distribution, and d(i,j) is the distance from pixel Pi to f. j The Euclidean distance of the edge, Map the input variable to between (0,1);

[0024] Through the aggregation function, the final rendering output I at pixel Pi i for:

[0025]

[0026] Where T is the pixel value on the texture map, T b Indicates the background color.

[0027] Furthermore, step S3 is specifically as follows:

[0028] Based on the generated rendered image with the same posture as the input image, calculate its pixel-level loss L compared with the input original image pix and the key point loss L lan ;

[0029] The pixel-level error is optimized based on the pixel value difference to optimize the lighting parameters. The expression is:

[0030] L pix =||I0-R(α,β,T,p0)||1

[0031] Where I0 represents the input original image, R represents the differentiable renderer, and R(α, β, T, p) is the rendered image generated by the renderer under random pose parameters p;

[0032] Key point loss uses a deep alignment network M to detect 68 key points in face images;

[0033] Input the original image and the rendered image into the deep alignment network to obtain the coordinates of 68 key points about the facial features. Calculate the point-to-point Euclidean distance between each key point to optimize the reconstruction result so that the posture of the rendered image remains consistent with that of the input image. The expression is:

[0034] L lan =||M(I0)-M(R(α,β,T,p0))||2

[0035] Among them, M() represents the input image to the deep alignment network, and obtains the coordinates of 68 key points about facial features.

[0036] Furthermore, step S4 is specifically as follows:

[0037] According to the generated rendering image with different input posture, calculate its generation adversarial loss L with the input image adv and identity consistency loss L id :

[0038] Consider this generation route as the generator part of the generative adversarial network, design a discriminator D to judge the authenticity of the image generated by the generator, and use the generative-adversarial mechanism to optimize the renderer to generate more realistic face images. The expression is:

[0039] L adv =logD(I0)+log(1-D(R(α,β,T,p)))

[0040] Among them, R(α, β, T, p) is the rendered image generated by the renderer under the random pose parameter p; D() represents the input discriminator, which determines whether the input image is a real image and obtains the probability of being a real image;

[0041] Train a face recognition network, extract 256-dimensional facial features of the input image and the rendered image, calculate the cosine distance as the similarity, and design an identity consistency loss function to determine whether the two images belong to the same person:

[0042] The identity consistency loss function expression is:

[0043] L id =1-cos(F R (R(α,β,T,p)),F R (I0))

[0044] Among them, cos() represents the cosine distance between two vectors, F R represents the face recognition network, F R () indicates inputting the image to the face recognition network to obtain 256-dimensional face features.

[0045] Furthermore, MTCNN is specifically used as a deep alignment network to input a face image and obtain the coordinates of 68 key points about facial features.

[0046] Furthermore, the face recognition network specifically adopts Light CNN-29v2.

[0047] Furthermore, facial features include eyes, eyebrows, nose, mouth and outline.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] 1. The method of the present invention introduces a truly differentiable renderer. Compared with an incompletely differentiable renderer, it can better utilize the gradient descent algorithm to learn the dense correspondence between two-dimensional images and three-dimensional faces, and obtain a higher quality three-dimensional face reconstruction result.

[0050] 2. For optimization-based methods, the input of three-dimensional face parameters is too abstract for neural networks, and the generated images often lack authenticity and identity consistency. The method of the present invention adds a branch generative adversarial network, which can generate images with the same identity as the target image but different postures. The introduced generative adversarial loss and identity consistency loss make the final optimized three-dimensional face result have higher authenticity and more detailed texture features. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the method of the present invention;

[0052] Figure 2 It is a network block diagram of the method of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0054] Example

[0055] like Figure 1 and Figure 2 As shown, a single-image 3D face reconstruction method based on a differentiable renderer comprises the following steps:

[0056] S1. Input the target face image into the trained regression network to obtain the initialized three-dimensional face parameters; in this embodiment, specifically:

[0057] The target face image is input into the pre-trained 3D face model, which adopts the currently common Large Scale Face Model, and the output is the initialized 3D face parameters, including identity parameter α, expression parameter β, texture parameter T and posture parameter p0, etc.

[0058] Among them, the expression of the 3DMM shape model is as follows:

[0059]

[0060] Where S represents the three-dimensional face shape, S represents the average face shape, and s α and β are the face identity and facial expression vector basis respectively, α is the 158-dimensional face identity coefficient, and β is the 29-dimensional facial expression coefficient.

[0061] S2, inputting the initialized 3D face parameters and posture parameters into a differentiable renderer, and obtaining a rendered image G1 with the same posture as the input image and a rendered image G2 with a different posture;

[0062] In this embodiment, G1 is rendered by the initialized 3D face parameters obtained in step 1, including identity parameter α, expression parameter β, texture parameter T and posture parameter p0. In the rendering process of G2, the input identity, expression and texture parameters are the same as those of the rendering process of G1, but the posture parameter p is obtained by random sampling, wherein the pitch and yaw angles are sampled from U(-40°, 40°), i.e., uniformly distributed, ranging from -40 degrees to 40 degrees; and the roll and tumble angles are sampled from U(-15°, 15°).

[0063] The differentiable renderer in this embodiment is different from the z-buffer renderer in the traditional rasterization rendering; currently, renderers based on three-dimensional meshes as input mostly use z-buffer rasterization renderers. The principle of the traditional rasterization renderer is based on the rules of computer graphics. The three-dimensional mesh is input, and a triangular face is first assigned to each pixel of the rendered image. Then, the various attributes (lighting, texture, normal vector) of the vertices in the triangular face corresponding to the pixel are interpolated to calculate the color of each pixel in the rendered image to obtain the rendered two-dimensional image. The traditional rasterization process is a discrete process and a non-differentiable process. In the research of the single-image three-dimensional reconstruction subject, the use of the traditional rasterization renderer will not be able to use the gradient descent algorithm to optimize the reconstruction parameters, resulting in the problem of low reconstruction quality for large postures (i.e. non-frontal, partially occluded) faces. The differentiable renderer of this embodiment considers the aggregation mechanism of the probability contribution of all triangles to handle occlusion, and the implementation steps are as follows:

[0064] Construct the probability map to estimate the triangle f j Probability contribution to pixel Pi

[0065]

[0066] in, σ is a scalar that controls the sharpness of the probability distribution, and d(i,j) is the distance from pixel Pi to f. j The Euclidean distance of the edge, Map the input variable to (0,1); in this example, σ is 0.01.

[0067] Through the aggregation function, the final rendering output I at pixel Pi i for:

[0068]

[0069] Where T is the pixel value on the texture map, T b Indicates the background color.

[0070] S3. Calculate the pixel-level loss L between the generated rendered image and the target face image based on the same pose as the input image. pix and the key point loss L lan ;

[0071] Pixel loss is optimized based on pixel value difference, which optimizes lighting parameters such as ambient color, direction, distance and color of light source, and helps to improve the restored texture features. The expression is:

[0072] L pix =||I0-R(α,β,T,p0)||1

[0073] Among them, I0 represents the target face image, and R represents the renderer;

[0074] The landmark loss uses a deep alignment network M to detect 68 key points on the face image. In this embodiment, MTCNN is used as the deep alignment network, the original image and the rendered image are input, and the coordinates of 68 key points about the facial features (eyes, eyebrows, nose, mouth, contour) are obtained. The point-to-point Euclidean distance between each key point is calculated to optimize the reconstruction result so that the rendered image maintains the same posture as the input image. The expression is:

[0075] L lan =||M(I0)-M(R(α,β,T,p0))||2.

[0076] Among them, M() represents the input image to the deep alignment network, and obtains the coordinates of 68 key points about facial features.

[0077] S4. Calculate the generation adversarial loss L between the generated rendered image and the input image based on the rendered image with different postures from the input image. adv and identity consistency loss L id :

[0078] The generation route is regarded as the generator part of the generative adversarial network. A discriminator D is designed. The discriminator is a general decoder network structure. The generated rendered image and the original image are input into the discriminator D to finally obtain a probability. The probability represents the probability that the image is a real image, which is used to judge the authenticity (real / fake) of the image generated by the generator. The generative-adversarial mechanism is used to optimize the renderer to generate a more realistic face image. The expression is:

[0079] L adv=logD(I0)+log(1-D(R(α,β,T,p)))

[0080] Among them, R(α, β, T, p) is the rendered image generated by the renderer under random posture parameters p, and D() represents the input discriminator, which determines whether the input image is a real image and obtains the probability of being a real image.

[0081] Among them, in the generative-adversarial mechanism, the training goal of the generator is to generate sufficiently realistic images, and the training goal of the discriminator is to more accurately judge the authenticity of the image. The generator and the discriminator are in a "game" relationship. In this embodiment and formula, the goal of the generator (renderer) is to make the generated rendered image under random posture parameters p "cheat" the discriminator, and finally train the discriminator so that it cannot distinguish the authenticity of the real image and the generated rendered image. This formula represents the design and optimization of the generator (renderer).

[0082] Train a face recognition network F R In this embodiment, Light CNN-29v2 is used as the face recognition network to extract 256-dimensional face features (Embeddings) of the input image and the rendered image, calculate the cosine distance between the two as the similarity, and design an identity consistency loss function to determine whether the two images belong to the same person:

[0083] The identity consistency loss function expression is:

[0084] L id =1-cos(F R (R(α,β,T,p)),F R (I0))

[0085] Among them, cos() represents the cosine distance between two vectors, F R represents the face recognition network, F R () indicates inputting the image to the face recognition network and obtaining 256-dimensional face features.

[0086] S5. Optimize the three-dimensional face parameters according to the calculated loss value, and return the updated three-dimensional face parameters to step S2 until the iteration converges to obtain the optimized three-dimensional face parameters.

[0087] It should also be noted that in this specification, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0088] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A single-image 3D face reconstruction method based on a differentiable renderer, characterized in that: The following steps are involved: S1. Input the target face image into the pre-trained 3D face model to obtain the initialized 3D face parameters; S2, inputting the initialized 3D face parameters and posture parameters into a differentiable renderer, and obtaining a rendered image with the same posture as the input image and a rendered image with a different posture; The differentiable renderer handles occlusion by taking into account the aggregation mechanism of the probability contribution of all triangles. The implementation steps are as follows: Construct a probability map to estimate the triangle f j Probability contribution to pixel Pi in, σ is a scalar that controls the sharpness of the probability distribution, and d(i,j) is the distance from pixel Pi to f. j The Euclidean distance of the edge, Map the input variable to between (0,1); Through the aggregation function, the final rendering output I at pixel Pi i for: S3, comparing the target face image with the rendered image in the same posture to obtain key point loss values ​​and pixel-level loss values; S4, comparing the target face image with the rendered images in different postures to obtain the generated adversarial loss value and the identity consistency loss value; S5. Optimize the three-dimensional face parameters according to the calculated loss value, and return the updated three-dimensional face parameters to step S2 until the iteration converges to obtain the optimized three-dimensional face parameters.

2. The single-image 3D face reconstruction method based on a differentiable renderer according to claim 1, characterized in that: Step S1 is specifically as follows: Input the target face image into the pre-trained Large Scale Face Model 3D face model, and regress to obtain the initialized 3D face parameters, including face identity coefficient α, face expression coefficient β, texture parameter T, and posture parameter p0; Among them, the expression of the three-dimensional face model is as follows: Where S represents the three-dimensional face shape, represents the average face shape, s α and β are face identity and facial expression vector basis respectively, α is the face identity coefficient, and β is the facial expression coefficient.

3. The single-image 3D face reconstruction method based on a differentiable renderer according to claim 2, characterized in that: The attitude parameters are obtained by random sampling, where the pitch and yaw angles are sampled from U(-40°, 40°), and the roll angle is sampled from U(-15°, 15°).

4. The single-image 3D face reconstruction method based on a differentiable renderer according to claim 3, characterized in that: The initialized 3D face parameters and posture parameters obtained by regression are input into a differentiable renderer to obtain a rendered image G1 with the same posture as the input image and a rendered image G2 with a different posture from the input image.

5. The single-image 3D face reconstruction method based on a differentiable renderer according to claim 4, characterized in that: Step S3 is specifically as follows: Based on the generated rendered image with the same posture as the input image, calculate its pixel-level loss L compared with the input original image pix and the key point loss L lan ; The pixel-level error is optimized based on the pixel value difference to optimize the lighting parameters. The expression is: L pix =||I0-R(α,β,T,p0)||1 Where I0 represents the input original image, R represents the differentiable renderer, and R(α, β, T, p0) is the rendered image generated by the renderer under the pose parameter p0; Key point loss uses a deep alignment network M to detect 68 key points in face images; Input the original image and the rendered image into the deep alignment network to obtain the coordinates of 68 key points about the facial features. Calculate the point-to-point Euclidean distance between each key point to optimize the reconstruction result so that the posture of the rendered image remains consistent with that of the input image. The expression is: L lan =||M(I0)-M(R(α,β,T,p0))||2 Among them, M() represents the input image to the deep alignment network, and obtains the coordinates of 68 key points about facial features.

6. The single-image 3D face reconstruction method based on a differentiable renderer according to claim 5, characterized in that: Step S4 is specifically as follows: Based on the generated rendered image with different pose from the input image, calculate its generation adversarial loss L with the input image adv and identity consistency loss L id : The generation route is regarded as the generator part of the generative adversarial network. A discriminator D is designed to judge the authenticity of the image generated by the generator. The generative adversarial mechanism is used to optimize the renderer to generate more realistic face images. The expression is: THE adv =logD(I0)+log(1-D(R(α,β,T,p))) Among them, R(α, β, T, p) is the rendered image generated by the renderer under the random pose parameter p; D() represents the input discriminator, which determines whether the input image is a real image and obtains the probability of being a real image; Train a face recognition network, extract 256-dimensional facial features of the input image and the rendered image, calculate the cosine distance as the similarity, and design an identity consistency loss function to determine whether the two images belong to the same person: The identity consistency loss function expression is: L id =1-cos(F R (R(α,β,T,p)),F R (I0)) Among them, cos() represents the cosine distance between two vectors, F R represents the face recognition network, F R () indicates inputting the image to the face recognition network and obtaining 256-dimensional face features.

7. The single-image 3D face reconstruction method based on a differentiable renderer according to claim 6, characterized in that: Specifically, MTCNN is used as the deep alignment network, and a face image is input to obtain the coordinates of 68 key points about facial features.

8. The single-image 3D face reconstruction method based on a differentiable renderer according to claim 7, characterized in that: The face recognition network specifically uses Light CNN-29v2.

9. The single-image 3D face reconstruction method based on a differentiable renderer according to claim 8, characterized in that: The facial features include eyes, eyebrows, nose, mouth and outline.

Images