A method for improving visual effects of a 3DMM face model

Abstract

The application discloses a method for improving the visual effect of a 3DMM face model. Existing methods ignore the fidelity of the face model in human visual perception, and the designed model and method do not match the subjective feeling of people. The method of the application first pre-processes and image normalizes a face data set, outputs 3DMM reconstruction coefficients and radiation compensation coefficients through a neural network, decodes a personalized face model through a Basel model, fuses the texture vector of the personalized face model with the radiation compensation coefficients, calculates a loss function, restricts the network learning direction, and optimizes the visual effect of the face model. The radiation intensity compensation module and the texture loss function added in the original method can improve the texture space range of the reconstructed face model. The method can effectively improve the expression deficiency of the 3DMM parameterized face model and improve the visual performance of the model under human visual perception.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer, especially the field of computer vision and three-dimensional face reconstruction, and particularly relates to a method for improving the visual effect of a 3DMM face model. BACKGROUND

[0002] The reconstruction of three-dimensional face geometry and texture has always been an important task in computer vision and graphics. There are many applications in virtual reality, cultural fields, and social media, such as virtual tour guides, live broadcasts, etc. The above task scenarios require that the face reconstruction meet the similarity of human visual sense. Due to the particularity of human visual sense, millimeter distance errors cannot be perceived, but color information is very sensitive, that is, the texture information on the face. However, past methods and research have always focused on improving the accuracy of face reconstruction, ignoring the fidelity of the face model in human visual sense, and the designed model and method do not match the subjective feeling of people.

[0003] Three-dimensional morphable model (3DMM) is a general statistical model of three-dimensional face shape and texture with explicit linear relationship, which is widely used in many aspects, such as face reconstruction, face registration, face recognition, etc. The advantage of this model is that it has strong prior knowledge, and by inputting different parameters, the same object can be reconstructed in different expressions. At the same time, the complex face reconstruction problem is optimized as a result of superimposing a single face by multiple face feature lines.

[0004] The 3DMM parameterized face model is learned by a training set of jointly collected three-dimensional face scans and 2D images. In order to obtain a high-precision three-dimensional face model, a large number of high-quality 3D face scans are usually required. However, it is difficult to obtain a large number of 3D face scans. The model is initially established by scanning 200 subjects with similar races / ages, which only contains neutral expressions, and the reconstructed face is easy to collapse under some expressions. The Basel Face Model (BFM) widely used at present is also established on the basis of 200 subjects with neutral expressions, and the model later improved the lack of expressions by using the public data sets FaceWarehouse and BD-3FE expression library to construct the linear basis of expressions, greatly improving the expression performance of the reconstructed three-dimensional face.

[0005] Parameterized face model uses neural network, increases constraint, mask, limits network learning area of picture, improves precision of model and generalization ability of model, and successfully constructs face model similar to target face. A representative method is to use 3DMM model, to constrain network learning by selecting a suitable loss function method, and to construct a fine face model end to end. However, the texture and shape of the 3DMM face model are limited by the space that can be expressed by the linear basis learned by the PCA method, and the low-dimensional representation method makes the model texture perform poorly in high-frequency details. Since three-dimensional face scanning is difficult to obtain, in order to obtain a more detailed three-dimensional face, scholars propose to use UV space as a representation method for face texture. UV space has no prior knowledge of face topology, and cannot accurately estimate the UV map of the face with face occlusion. At the same time, an additional encoding-decoding network needs to be constructed for mapping, which has a complex structure and poor robustness. In addition, the neural radiance field method has recently been widely concerned by scholars, and its reconstruction effect can be amazing, but it needs spatial coordinates and camera parameters as input, the pipeline is complex, and the rendering result of the final model is seriously dependent on the training set, the training time is long, and the generalization performance is poor. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a method for improving the visual effect of a 3DMM face model. When using a 3DMM parameterized model as a face generation tool, the visual performance of the final face model is improved to solve the problem of insufficient expressiveness of the parameterized model.

[0007] The method of the present application is as follows:

[0008] Step (1) pre-processes the face data set, including:

[0009] The MTCNN face detection module is used to extract 5 face key points from each initial face picture in the face data set;

[0010] Each initial face picture is cropped to a 224x224 size face picture according to the corresponding 5 face key points;

[0011] The cropped face picture is detected for 68 face key points by using a pre-trained face detection model;

[0012] The Prewitt operator built-in OpenCV is used to extract the high-frequency information map T of each cropped face picture ex Further, specifically: according to the sum P(i,j) of the extracted horizontal information and vertical information, it is determined whether the pixel point (i,j) is an edge, if P(i,j) is greater than a set threshold, it indicates that the pixel point is an edge pixel point, the edge pixel point is retained, and other pixel points are filtered out, to obtain the high-frequency information map T exP(i,j) = G(i) + G(j); G(i) and G(j) represent the value of the extracted horizontal edge information and the value of the extracted vertical edge information, respectively;

[0013] G(i) = |[f(i-1,j-1) + f(i-1,j) + f(i-1,j+1)] - [f(i+1,j-1) + f(i+1,j) + f(i+1,j+1)]|,

[0014] G(j) = |[f(i-1,j+1) + f(i,j+1) + f(i+1,j+1)] - [f(i-1,j-1) + f(i,j-1) + f(i+1,j-1)]|;

[0015] f(i,j) represents the pixel value of pixel point (i,j);

[0016] The Bayesian selector is used to perform face skin judgment on each face picture pixel by pixel to obtain a face mask M. Further, the probability value of the nth pixel in the face mask M being judged as skin is P n Pn represents the probability of the Bayesian selector judging the nth pixel of the face picture as skin, n = 1, 2, …, N, and N is the number of pixel points of the face picture.

[0017] Step (2) image normalization processing: the three-channel pixel value range of each cropped face picture I is compressed from the 0-255 interval to the 0-1 interval, and the single-channel pixel value range of the corresponding high-frequency information map T ex is compressed from the 0-255 interval to the 0-1 interval.

[0018] Step (3) inputting the normalized face picture into the ResNet50 neural network to output two groups of coefficients, which are 3DMM reconstruction coefficients and radiation compensation coefficients where (α, β, γ) is the 3DMM parameterization model coefficient, is the illumination coefficient, is the pose coefficient, represents the real number field, 239 is the total number of coefficients, and 3x36x10 3 is the dimension of the texture vector analyzed by the 3DMM parameterization model. The two groups of coefficients are regressed using the same basic network, and finally two different fully connected layers are used for coefficient regression.

[0019] Further, the pose coefficient P is composed of a rotation coefficient and a translation coefficient, the rotation coefficient R ∈ SO(3), SO(3) represents a third-order rotation matrix, and the translation coefficient

[0020] Step (4) inputs the regression 3DMM parameterized model coefficients (α,β,γ) into the Basel model for analysis, and decodes the low-dimensional personalized face model, which includes shape vector S and texture vector T.

[0021] further, and For the average face geometry and average face texture derived from the Basel model, B id B exp and B t These are the principal components extracted from identity, facial expression, and texture information by the Basel model.

[0022] Step (5) combines the texture vector T of the low-dimensional personalized face model with the radiation compensation coefficient. For fusion, the texture vector T = (R′1, G′1, B′1, R′2, G′2, B′2, ..., R′) 36000 ,G′ 36000 ,B′ 36000 ), (R′ m ,G′ m ,B′ m ( ) represents the RGB information of the m-th vertex in the personalized face model, where m = 1, 2, ..., 36000. (R″ m ,G″ m ,B″ m ( ) represents the RGB compensation information of the m-th vertex in the personalized face model, where the texture vector T and the radiation compensation coefficient are... Linear overlay is performed, and texture information is compensated vertex-by-vertex to obtain the radiation intensity of a personalized face model.

[0023] Step (6) Obtain a 2D projection image I′ of a personalized face model consistent with face image I using a differentiable renderer, and calculate the loss function; specifically:

[0024] The shape vector S and the compensated texture vector T are used as the reconstructed face model vector. The reconstructed face model vector, along with the illumination coefficient and pose coefficient, are input into the differentiable renderer. The differentiable renderer bakes a visualized 3D face model that is similar to the face image.

[0025] The differentiable renderer adjusts the camera pose according to the pose coefficients, performs a two-dimensional projection on the generated visualized 3D face model, and obtains a two-dimensional projection image I′ that is consistent with the face image I;

[0026] The loss function is calculated by comparing the 2D projected image I′ obtained through a differentiable renderer with the face image I, and the objective function is L. total =L image +Llmk +L tex +L P +L coeff ; wherein, L image is the luminosity loss, L lmk is the key point loss, L tex is the high-frequency texture loss, L P is the high-dimensional feature cosine distance, L coeff is the regularization loss; the loss function is used to constrain the learning direction of the network, reduce the difference between the two-dimensional projection picture I' and the face picture I, and optimize the visual effect of the reconstructed face model.

[0027] The original method and research have always been to improve the face reconstruction accuracy as the working direction, ignoring the fidelity of the face model in the human eye visual sense, and the designed model and method do not match the subjective feeling of people. The present application considers the deficiencies of the past work and the constraint of the 3DMM parameterized model as a linear model on the basis of the original 3DMM parameterized model, adds a radiation intensity compensation module and a texture loss function to improve the texture space range of the reconstructed face model. The method of the present application can effectively improve the problem of insufficient expressiveness of the 3DMM parameterized face model and improve the visual performance of the model under the human eye sense. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the overall flowchart of the present application;

[0029] Figure 2 is the structural framework schematic diagram of the method of the present application;

[0030] Figure 3 is the feature map flowchart of the face picture preprocessing extraction. DETAILED DESCRIPTION

[0031] The technical solutions of the present application are further described below in combination with specific embodiments:

[0032] As shown in Figure 1 and 2 , a method for improving the visual effect of a 3DMM face model is as follows:

[0033] Step (1) pre-processes the face data set, as shown in Figure 3 , specifically including:

[0034] Using the MTCNN face detection module to extract 5 face key points from each initial face picture in the face data set;

[0035] Cutting out a 224x224 size face picture from each initial face picture according to the corresponding 5 face key points;

[0036] The cropped face picture is detected by a pre-trained face detection model to obtain 68 face key points;

[0037] The high-frequency information graph T of each cropped face picture is extracted by using the built-in Prewitt operator of OpenCV ex ;

[0038] According to the sum P(i,j) of the extracted horizontal information and vertical information, it is determined whether the pixel point (i,j) is an edge, if P(i,j) is greater than a set threshold, it indicates that the pixel point is an edge pixel point, the edge pixel point is retained, and other pixel points are filtered out to obtain the high-frequency information graph T ex ; P(i,j)=G(i)+G(j); G(i) and G(j) represent the values extracted by the horizontal edge information and the vertical edge information, respectively;

[0039] G(i)=|[f(i-1,j-1)+f(i-1,j)+f(i-1,j+1)]-[f(i+1,j-1)+f(i+1,j)+f(i+1,j+1)]|,

[0040] G(j)=|[f(i-1,j+1)+f(i,j+1)+f(i+1,j+1)]-[f(i-1,j-1)+f(i,j-1)+f(i+1,j-1)]|;

[0041] f(i,j) represents the pixel value of the pixel point (i,j).

[0042] The Bayesian selector trained by the face database is used to perform face skin judgment on each face picture pixel by pixel to obtain a face mask M; P n represents the probability that the Bayesian selector judges the nth pixel of the face picture as skin, A n represents the probability value that the nth pixel in M is judged as skin, n=1,2,…,N, and N is the number of pixel points of the face picture.

[0043] Step (2) image normalization processing: the three-channel pixel value range of each cropped face picture I is compressed from the interval 0-255 to the interval 0-1, and the single-channel pixel value range of the corresponding high-frequency information graph T ex is compressed from the interval 0-255 to the interval 0-1.

[0044] Step (3) input the normalized face picture into the ResNet50 neural network, and output two groups of coefficients after neural calculation, which are 3DMM reconstruction coefficients and radiation compensation coefficients , wherein (α,β,γ) is the 3DMM parameterized model coefficient, This is the illumination coefficient. For attitude coefficients, Represents the real number field, 239 is the total coefficient, 3 × 36 × 10 3 The dimension of the texture vector parsed from the 3DMM parametric model.

[0045] The attitude coefficient P consists of rotation coefficients and translation coefficients, where the rotation coefficient R ∈ SO(3), SO(3) represents a third-order rotation matrix, and the translation coefficients are...

[0046] Both sets of coefficient regressions use the same base network, and finally use two different fully connected layers for coefficient regression.

[0047] Step (4) Input the regression 3DMM parameterized model coefficients (α,β,γ) into the Basel model for analysis. The Basel model is a classic 3DMM parameterized face model, which decodes a low-dimensional personalized face model, including two parts: shape vector S and texture vector T. and For the average face geometry and average face texture derived from the Basel model, B id B exp and B t These are the principal components extracted from identity, facial expression, and texture information by the Basel model.

[0048] Step (5) combines the texture vector T of the low-dimensional personalized face model with the radiation compensation coefficient. For fusion, the texture vector T = (R′1, G′1, B′1, R′2, G′2, B′2, ..., R′) 36000 ,G′ 36000 ,B′′ 6000 ), (R′ m ,G′ m ,B′ m ( ) represents the RGB information of the m-th vertex in the personalized face model, where m = 1, 2, ..., 36000. (R″ m ,G″ m ,B″ m ( ) represents the RGB compensation information of the m-th vertex in the personalized face model, where the texture vector T and the radiation compensation coefficient are... Linear overlay is performed, and texture information is compensated vertex-by-vertex to obtain the radiation intensity of a personalized face model.

[0049] Step (6) Obtain a 2D projection image I′ of a personalized face model consistent with face image I using a differentiable renderer, and calculate the loss function; specifically:

[0050] The shape vector S and the compensated texture vector T are taken as a reconstructed face model vector, and the reconstructed face model vector is input into a differentiable renderer together with illumination coefficients and pose coefficients, and the differentiable renderer bakes a visual three-dimensional face model similar to the face picture;

[0051] The differentiable renderer adjusts the camera pose according to the pose coefficients, performs two-dimensional projection on the generated visual three-dimensional face model, and obtains a two-dimensional projection picture I' consistent with the face picture I;

[0052] The two-dimensional projection picture I' obtained through the differentiable renderer is subjected to loss function calculation with the face picture I, and the objective function L total = L image + L lmk + L tex + L P + L coeff ; wherein L image is a luminosity loss, L lmk is a key point loss, L tex is a high-frequency texture loss, L P is a high-dimensional feature cosine distance, and L coeff is a regularization loss. The learning direction of the network is constrained by using the loss function, the gap between the two-dimensional projection picture I' and the face picture I is reduced, and the visual effect of the reconstructed face model is optimized.

Claims

1. A method for improving the visual effect of 3DMM face models, characterized in that: Step (1) Preprocesses the face dataset, including: The MTCNN face detection module was used to extract five facial landmarks from each initial face image in the face dataset. Each initial face image is cropped into a 224×224 size face image based on the corresponding 5 facial key points; The cropped face image was used to detect 68 facial key points using a pre-trained face detection model; The high-frequency information map T of each cropped face image was extracted using OpenCV's built-in Prewitt operator. ex ; The Bayesian selector is used to perform facial skin detection pixel by pixel on each face image to obtain the facial mask M; Step (2) Image normalization processing: Compress the three-channel pixel value range of each cropped face image I from the range of 0 to 255 to the range of 0 to 1, and normalize the corresponding high-frequency information image T. ex The range of single-channel pixel values ​​has been compressed from 0 to 255 to 0 to 1. Step (3) Input the normalized face image into the ResNet50 neural network, and output two sets of coefficients, namely the 3DMM reconstruction coefficients. and radiation compensation coefficient Where (α,β,γ) are the coefficients of the 3DMM parametric model. This is the illumination coefficient. For attitude coefficients, Represents the real number field, 239 is the total coefficient, 3 × 36 × 10 3 The dimension of the texture vector parsed from the 3DMM parametric model; Both sets of coefficient regressions use the same base network, and finally use two different fully connected layers for coefficient regression; Step (4) inputs the regression 3DMM parameterized model coefficients (α,β,γ) into the Basel model for analysis, and decodes the low-dimensional personalized face model, which includes shape vector S and texture vector T; Step (5) combines the texture vector T of the low-dimensional personalized face model with the radiation compensation coefficient. For fusion, the texture vector T = (R′1,G′1,B′1,R′2,G′2,B′2,…,R′) 36000 ,G′ 36000 ,B′ 36000 ), (R′ m ,G′ m ,B′ m ( ) represents the RGB information of the m-th vertex in the personalized face model, where m = 1, 2, ..., 36000. (R″ m ,G″ m ,B″ m ( ) represents the RGB compensation information of the m-th vertex in the personalized face model, where the texture vector T and the radiation compensation coefficient are... Linear overlay is performed, and texture information is compensated vertex-by-vertex to obtain the radiation intensity of a personalized face model. Step (6) Obtain a 2D projection image I′ of a personalized face model consistent with face image I using a differentiable renderer, and calculate the loss function; specifically: The shape vector S and the compensated texture vector T are used as the reconstructed face model vector. The reconstructed face model vector, along with the illumination coefficient and pose coefficient, are input into the differentiable renderer. The differentiable renderer bakes a visual 3D face model that is similar to the face image. The differentiable renderer adjusts the camera pose according to the pose coefficients, performs a two-dimensional projection on the generated visualized 3D face model, and obtains a two-dimensional projection image I′ that is consistent with the face image I; The loss function is calculated by comparing the 2D projected image I′ obtained through a differentiable renderer with the face image I, and the objective function is L. total =L image +L lmk +L tex +L P +L coeff ; among which, L image For luminance loss, L lmk For critical point loss, L tex For high-frequency texture loss, L P L is the high-dimensional feature cosine distance. coeff The loss function is used to constrain the learning direction of the network, reduce the gap between the 2D projected image I′ and the face image I, and optimize the visual effect of the reconstructed face model.

2. The method for improving the visual effect of a 3DMM face model as described in claim 1, characterized in that: Extracting high-frequency information graph T in step (1) ex The specific method is as follows: Pixel (i,j) is determined to be an edge based on the sum of extracted horizontal and vertical information, P(i,j). If P(i,j) is greater than a set threshold, the pixel is considered an edge pixel, and it is retained while other pixels are filtered out, resulting in a high-frequency information map T. ex P(i,j) = G(i) + G(j); G(i) and G(j) represent the values ​​of the extracted horizontal edge information and the extracted vertical edge information, respectively. G(i)=|[f(i-1,j-1)+f(i-1,j)+f(i-1,j+1)]-[f(i+1,j-1)+f(i+1,j)+f(i+1,j+1)]|, G(j)=|[f(i-1,j+1)+f(i,j+1)+f(i+1,j+1)]-[f(i-1,j-1)+f(i,j-1)+f(i+1,j-1)]|; f(i,j) represents the pixel value of pixel (i,j).

3. The method for improving the visual effect of a 3DMM face model as described in claim 1, characterized in that: Step (1) The probability value of the nth pixel in the face mask M being identified as skin. P n This represents the probability that the Bayesian selector identifies the nth pixel of a face image as skin, where n = 1, 2, ..., N, and N is the number of pixels in the face image.

4. The method for improving the visual effect of a 3DMM face model as described in claim 1, characterized in that: The attitude coefficient P mentioned in step (3) consists of rotation coefficients and translation coefficients. The rotation coefficient R ∈ SO(3), where SO(3) represents a third-order rotation matrix, and the translation coefficient R ∈ SO(3).

5. The method for improving the visual effect of a 3DMM face model as described in claim 1, characterized in that: In step (4), and For the average face geometry and average face texture derived from the Basel model, B id B exp and B t These are the principal components extracted from identity, facial expression, and texture information by the Basel model.

Images