Human face three-dimensional model reconstruction method, computer device and storage medium
By acquiring and classifying face base data, calculating the fit weight, and combining the feature points of the face image, the complete three-dimensional face reconstruction of a single face image is achieved, solving the complexity and accuracy of three-dimensional face reconstruction in the existing technology. The problem of sexual inadequacy meets the needs of personalized facial mask design.
Patent Information
- Application Number
- CN202311711410.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-12
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2043-12-12
AI Technical Summary
The prior art is difficult to meet the needs of three-dimensional reconstruction of the complete face surface when performing three-dimensional reconstruction based on a single face image, and the calculation volume is large and complex, so it cannot meet the needs of personalized facial mask design and other fields.
By obtaining multiple face base data, classifying and calculating the average face data, width-length ratio and face-shaped area ratio, determining the fitting weight, combining the feature points of the face image to be processed and the feature points of the base data, fitting a three-dimensional reconstruction of the face. The model considers features such as forehead, eyebrows, eyes, nose, lips and chin to achieve a comprehensive reconstruction of a three-dimensional face.
Complete three-dimensional face reconstruction based on a single face image is realized, meeting the needs of personalized facial mask design and other scenarios, and improving the accuracy and efficiency of three-dimensional face reconstruction.
Smart Images

Figure CN117876577B_ABST
Abstract
Description
Technical Field
[0001] The present embodiment relates to the field of image processing technology, and in particular to a method for reconstructing a three-dimensional face model, a computer device, and a storage medium. Background Art
[0002] In the field of information data processing, 3D face reconstruction based on 2D image data is widely used in face recognition, expression animation, etc. For example, when designing personalized facial masks, due to the limited image acquisition capabilities of scenes such as stores, outdoor or self-service equipment, generally only 2D image data of the face can be collected, and 3D face reconstruction is required to obtain a 3D face, so as to design the shape of the mask.
[0003] Some current face 3D model reconstruction technologies generally require collecting multiple face images of a person and reconstructing based on the multiple face images. Although multiple images can better restore the depth information of the face, the alignment and comprehensive calculations are large and the reconstruction method is cumbersome. Although other face 3D model reconstruction technologies can perform real-time 3D face reconstruction based on a single face image, it is easy to lose face surface information when collecting a single face image, and cannot meet the needs of complete face surface 3D reconstruction. Summary of the invention
[0004] In view of the technical problems that the current face three-dimensional model reconstruction technology is difficult to meet the needs of three-dimensional reconstruction of the complete face surface when reconstructing based on a single face image, the purpose of this embodiment is to provide a face three-dimensional model reconstruction method, computer device and storage medium.
[0005] On the one hand, the present embodiment includes a method for reconstructing a 3D face model, and the method for reconstructing a 3D face model includes the following steps:
[0006] Acquire multiple facial base data;
[0007] Classifying each of the facial base data according to the gender and face shape of the person from whom it was collected, so that each of the facial base data is classified into a corresponding facial category;
[0008] Calculate the average face data, the average estimated value of the width-to-length ratio and the average estimated value of the face shape area ratio corresponding to each of the face categories according to the face base data;
[0009] Obtain the face image to be processed;
[0010] Determining fitting weights according to feature points corresponding to the face image to be processed and feature points corresponding to the face base data;
[0011] According to the fitting weights, each of the face base data is fitted to obtain a three-dimensional face reconstruction model corresponding to the face image to be processed.
[0012] Furthermore, the acquiring of a plurality of face base data includes:
[0013] Select multiple collection source personnel; each of the collection source personnel corresponds to a corresponding gender and face shape; the face shape is any one of the following: Tian face, You face, Shen face, Guo face, Jia face, and Mu face;
[0014] Under a constant environment, use a three-dimensional laser scanning device to scan each of the acquisition source personnel to obtain a three-dimensional facial surface model; the three-dimensional facial surface model includes a plurality of vertices;
[0015] The facial three-dimensional surface model is normalized, and dense correspondence is established between vertices in the facial three-dimensional surface model and the face of the acquisition source person to obtain the facial base data.
[0016] Furthermore, the calculating of the average face data, the average estimated value of the width-to-length ratio and the average estimated value of the face area ratio corresponding to each of the face categories according to the face base data includes:
[0017] Performing cylindrical coordinate transformation on the face base data to obtain a two-dimensional frontal face photo;
[0018] According to the formula and Calculate the face width-to-length ratio α and the face area ratio β corresponding to the two-dimensional frontal face photo, where W and h are respectively the width and height of the rectangular area where the face is located in the two-dimensional frontal face photo, and s is the area of the face area in the two-dimensional frontal face photo;
[0019] Get the face width-to-length ratios corresponding to all n two-dimensional frontal photos of the face corresponding to the i-th gender and the j-th face shape, according to the formula Calculate the average estimate of the aspect ratio where α k represents the face width-to-length ratio corresponding to the kth two-dimensional frontal face photo among all the n two-dimensional frontal face photos;
[0020] Get the face area ratios corresponding to all n two-dimensional frontal photos of the face corresponding to the i-th gender and the j-th face shape, according to the formula Calculate the average estimated value of the face area ratio where β k It represents the face area ratio corresponding to the kth two-dimensional frontal face photo among all the n two-dimensional frontal face photos.
[0021] Furthermore, the facial base data are fitted according to the fitting weights to obtain a 3D facial reconstruction model corresponding to the facial image to be processed, and the formula used is:
[0022]
[0023] Where S is the 3D reconstruction model of the face corresponding to the face image to be processed, w j is the fitting weight corresponding to the facial base data of the j-th face shape, m is the total number of facial base data corresponding to each gender, is the average face data corresponding to the face base data of the i-th gender and the j-th face shape, is the average face data corresponding to the face base data of the i-th gender.
[0024] Furthermore, determining the fitting weights according to the feature points corresponding to the face image to be processed and the feature points corresponding to the face base data includes:
[0025] Get the feature point matrix M corresponding to the face image to be processed 2d ;
[0026] According to the feature point matrix M corresponding to the face image to be processed 2d The feature point matrix M corresponding to the face base data 3d , performing facial posture estimation on the face image to be processed to obtain a rotation matrix R and a displacement vector T;
[0027] According to the formula
[0028]
[0029]
[0030] Determine the matrix A and matrix b; where, is the average face data corresponding to all the face base data;
[0031] According to the formula w j+1 =w j +2·(Aw-b)A j Perform iterations to calculate the fitting weight w j ; where w={w1,w2,…,w m},
[0032] Furthermore, the feature point matrix M corresponding to the face image to be processed is obtained 2d ,include:
[0033] Using the Canny edge detection algorithm, the feature edge of the face image to be processed is located to determine the edge position in the face image to be processed;
[0034] Select feature points from the edge positions to form a matrix M 2d .
[0035] Furthermore, the feature point matrix M corresponding to the face image to be processed 2d The feature point matrix M corresponding to the face base data 3d , performing face pose estimation on the face image to be processed to obtain a rotation matrix R and a displacement vector T, including:
[0036] By using the least squares method to solve the formula Minimize and determine the rotation matrix R and the displacement vector T;
[0037] in, f x is the length of the face image to be processed, f y is the width of the face image to be processed, c x and c y is the center coordinate of the face image to be processed.
[0038] Furthermore, the iteration is performed according to the formula to calculate the fitting weight w j ,include:
[0039] Obtaining a face width-to-length ratio α and a face shape area ratio β of the face image to be processed;
[0040] According to the formula Set the fitting weight w j The initial value of is the average estimated value of the width-to-length ratio of all the face categories, is an average estimated value of the face area ratio for all the various face categories;
[0041] According to the set initial value, according to the formula w j+1 =w j +2·(Aw-b)A j Perform gradient descent iterations to calculate the fitting weight w j .
[0042] On the other hand, the present embodiment also includes a computer device, including a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute a method for reconstructing a three-dimensional face model in the embodiment.
[0043] On the other hand, the present embodiment further includes a storage medium storing a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to execute a method for reconstructing a three-dimensional face model in the embodiment.
[0044] The beneficial effects of the present embodiment are as follows: the method for reconstructing a three-dimensional face model in the embodiment determines the fitting weights according to the feature points corresponding to the face image to be processed and the feature points corresponding to the face base data, and can call the face base data of different face categories related to gender and face shape for the face image to be processed, and obtain the three-dimensional reconstruction model of the face by fitting the face base data, thereby realizing the reconstruction of the three-dimensional face model based on a face image to be processed; when the face base data includes features such as the forehead, eyebrows, eyes, nose, lips and chin, the features of the face can be considered more comprehensively and completely in the process of constructing the three-dimensional face, so that the corresponding three-dimensional face surface information can be restored according to a single face image to be processed, meeting the needs of scenarios such as personalized facial mask design. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 A schematic diagram of the steps of the method for reconstructing a three-dimensional face model in an embodiment;
[0046] Figure 2 A schematic diagram of the principle of extracting the face area in a two-dimensional frontal face photo in an embodiment;
[0047] Figure 3 Schematic diagram of the principle of extracting facial feature points in the embodiment. DETAILED DESCRIPTION
[0048] In this embodiment, it is considered that the current related technologies have the following disadvantages:
[0049] 1. Commonly used facial feature points focus on the recognition of facial expressions, so the eyes, nose, mouth and chin are mainly considered, but the feature points and surface reconstruction of the forehead area are lacking, resulting in insufficient practicality in areas such as personalized mask design, and unable to meet the needs of complete facial 3D reconstruction, especially in areas such as personalized mask design;
[0050] 2. To reconstruct a 3D face model based on a single face photo, a 3D face model database is needed. However, in the construction process of commonly used databases, only the distinction between male and female genders is generally considered, and the characteristics of different face shapes are not considered. The 3D deformation method of the face is not designed in combination with the facial features. The deformation method is complicated, the amount of calculation is large, and the effect is average;
[0051] 3. The estimation of facial posture is often calculated using a standard 3D face mesh model, such as the 3D average face model in the face database, without considering the features of the specific face. The original face database only has the 3D model of the face and the calculated 3D average face model, but does not have the feature information of the face. It is impossible to combine the 2D face features to be reconstructed to construct a personalized initial 3D face model for facial posture estimation, resulting in low accuracy in facial posture estimation.
[0052] Shortcoming 1 can be solved by the following ways: design a facial feature point model including forehead, eyebrows, eyes, nose, lips and chin, and comprehensively consider the facial features in the process of three-dimensional face construction; at the same time, in the construction of the face database, collect complete facial surface data including forehead and perform dense correspondence of different faces, and match it with the designed facial feature point model to provide a basis for constructing a complete facial surface model.
[0053] Shortcoming 2 can be solved in the following way: divide the face shape into 6 types according to facial features, namely, Tian face, You face, Shen face, Guo face, Jia face, and Mu face; build a database based on face shape according to gender, and calculate the average 3D face model of different face shapes respectively; and quickly fit the face surface model to be reconstructed through the weighted combination of these 6 average 3D face models.
[0054] Disadvantage 3 can be solved in the following way: based on the average width-to-length ratio and average face area ratio of various face shapes, obtain the width-to-length ratio and face area ratio of the two-dimensional face photo, calculate the similarity between the face to be reconstructed and various face shapes, and quickly form a personalized initial three-dimensional face model for estimating face posture.
[0055] Based on the above principle, in this embodiment, a method for reconstructing a three-dimensional face model is provided. Figure 1 ,The 3D face model reconstruction method includes the following steps:
[0056] S1. Obtain multiple face base data;
[0057] S2. Classify each facial base data according to the gender and face shape of the source person, so that each facial base data is classified into a corresponding face category;
[0058] S3. Calculate the average face data, average estimated value of width-to-length ratio and average estimated value of face area ratio corresponding to each face category according to the face base data of each face category;
[0059] S4. Obtaining the face image to be processed;
[0060] S5. Determine the fitting weights according to the feature points corresponding to the face image to be processed and the feature points corresponding to the face base data;
[0061] S6. Fit each face base data according to the fitting weight to obtain a face 3D reconstruction model corresponding to the face image to be processed.
[0062] In steps S1-S6, steps S1-S3 are steps for collecting facial base data and constructing a three-dimensional face database, and steps S4-S6 are steps for fitting a three-dimensional face reconstruction model.
[0063] The human face shape is mainly affected by the jaw, frontal bone, parietal bone, etc., and the jaw is the basic structure that determines the face shape. The face shape can be roughly divided into 6 types: Tian face (round face, flat and square), You face (narrow forehead, wide temples and chin), Shen face (narrow upper and pointed lower), Guo face (square, long face with wide forehead and cheeks), Jia face (pointed upper and lower), and Mu face (long face, narrow and long head). Therefore, when executing step S1, that is, the step of obtaining multiple facial base data, the following steps are specifically performed:
[0064] S101. Select multiple collection source personnel;
[0065] S102. In a constant environment, use a 3D laser scanning device to scan each source of the acquisition personnel to obtain a 3D facial surface model; the 3D facial surface model includes multiple vertices;
[0066] S103. Normalize the three-dimensional facial surface model, establish a dense correspondence between the vertices in the three-dimensional facial surface model and the face of the source person, and obtain facial base data.
[0067] When executing step S101, 600 males and females aged between 20 and 60 can be selected through voluntary recruitment, including 100 people of each gender with six face types, namely, Tianzi face, Youzi face, Shenzi face, Guozi face, Jiazi face and Muzi face, as the source personnel for collection, for a total of 1,200 people. Among these 1,200 people, there are male personnel with Tianzi face, female personnel with Tianzi face, male personnel with Youzi face, female personnel with Youzi face, etc.
[0068] When executing step S102, a complete three-dimensional surface model of a person's face can be collected using a three-dimensional laser scanning device under a constant environment, and then the three-dimensional face model data is normalized so that the number of vertices of the three-dimensional faces is the same and there is dense correspondence between different faces.
[0069] When executing step S103, in order to normalize the three-dimensional face model, the three-dimensional face model data can be first parameterized according to the three-dimensional cylindrical coordinates. The method is to set the central axis of the face in the vertical direction as the Z axis, the center of the face as the coordinate origin, the coordinate axis pointing to the front of the face as the X axis, and the Y axis perpendicular to the plane determined by the X axis and the Z axis.
[0070] The corresponding transformation between the three-dimensional coordinate system and cylindrical coordinates of each vertex in the three-dimensional face model is as follows:
[0071]
[0072] Where (x, y, z) are the vertex coordinates of the 3D face model, and (h, r, θ) are the cylindrical coordinates.
[0073] For the transformed cylindrical coordinates (h, r, θ), h and θ are used as X and Y coordinates in a plane rectangular coordinate system, and the three-dimensional face model data is transformed into a two-dimensional image.
[0074] When executing step S103, the face is divided into blocks according to the features of the face, such as dividing the face into several blocks with the same features according to the areas where the eyebrows, eyes, nose, mouth, forehead, chin, cheeks, etc. are located, and numbering them in sequence so that all face data correspond in area.
[0075] When executing step S103, in order to establish a dense correspondence of the face model, the same vertices can be resampled for each area of the three-dimensional face data, and the set of these vertices is called a template. The average length and width of the area corresponding to the two-dimensional image after all face cylinder transformations are selected as the size of the template. The number of vertices in each area is determined, and the dense correspondence of the vertices can be determined by performing longitudinal and lateral interpolation on each block, inserting new vertices corresponding to the template vertices, and deleting vertices that do not correspond to the template, so that the vertices of the three-dimensional face model data correspond one to one.
[0076] By executing step S1, each facial base data obtained corresponds to a source person, and the gender of the source person is one of male and female, and the face shape is one of the six face shapes of Tian face, You face, Shen face, Guo face, Jia face, and Mu face. Therefore, each facial base data can be classified according to the gender and face shape of the corresponding source person. After classification, each facial base data corresponds to a face category, that is, the face category corresponding to a facial base data indicates which gender and face shape the facial base data corresponds to. Therefore, when executing step S2, a facial base data can be represented as S ij , which means that the face category corresponding to this face base data is the i-th gender and the j-th face type, that is, the gender of the source person corresponding to this face base data is the i-th gender, and the face type of the corresponding source person is the j-th face type. Specifically, i=1 indicates male, i=2 indicates female, and j=1, 2, 3, 4, 5, 6 indicate Tian-shaped face, You-shaped face, Shen-shaped face, Guo-shaped face, Jia-shaped face, and Mu-shaped face, respectively.
[0077] Thus, for the face base data of the i-th gender and the j-th face type, the average face data corresponding to It can be calculated by the following formula:
[0078]
[0079] Wherein, n represents the number of source personnel corresponding to each face shape, and in this embodiment, n=100.
[0080] For the average face data corresponding to the face base data of the i-th gender It can be calculated by the following formula:
[0081]
[0082] Wherein, m represents the number of source personnel of each gender, and in this embodiment, m=600.
[0083] In this embodiment, when executing step S3, that is, calculating the average face data, average estimated value of width-to-length ratio and average estimated value of face area ratio corresponding to each face category according to each face base data, the following steps are specifically performed:
[0084] S301. Perform cylindrical coordinate transformation on the facial base data to obtain a two-dimensional frontal face photo;
[0085] S302. According to the formula and Calculate the face width-to-length ratio α and the face area ratio β corresponding to the two-dimensional frontal face photo, where W and h are respectively the width and height of the rectangular area where the face is located in the two-dimensional frontal face photo, and s is the area of the face area in the two-dimensional frontal face photo;
[0086] S303. Obtain the face width-to-length ratios of all n two-dimensional frontal photos corresponding to the i-th gender and the j-th face shape, according to the formula Calculate the average estimate of the aspect ratio where α k Indicates the face width-to-length ratio of the kth 2D frontal face photo among all n 2D frontal face photos;
[0087] S304. Obtain the face area ratios corresponding to all n two-dimensional frontal photos of the face corresponding to the i-th gender and the j-th face shape, according to the formula Calculate the average estimate of the face area ratio where β k It represents the face area ratio corresponding to the kth two-dimensional frontal face photo among all n two-dimensional frontal face photos.
[0088] When executing step S301, the three-dimensional facial base data [specifically, it can be expressed as vertex coordinates (x, y, z)] can be transformed into cylindrical coordinates (h, r, θ) according to formula (1), and h and θ are taken as the x and Y coordinates in the plane rectangular coordinate system, thereby transforming the three-dimensional facial base data into a two-dimensional frontal facial photo.
[0089] When executing step S301, the 1200 persons may also be photographed using a digital camera in a constant environment to obtain two-dimensional frontal facial photographs.
[0090] In step S302, refer to Figure 2 , extract the rectangular area where the face is located in the two-dimensional frontal face photo, and the area where the face itself is located, detect these areas, and obtain the width W and height h of the rectangular area where the face is located in the two-dimensional frontal face photo, as well as the area s of the area where the face is located in the two-dimensional frontal face photo.
[0091] In step S303, the following formula is used for calculation:
[0092]
[0093] Thus, the face width-to-length ratio α and the face area ratio β corresponding to the two-dimensional frontal face photo are obtained.
[0094] For a specific 2D frontal face photo, the face width-to-length ratio α and face area ratio β calculated by formula (4) can be marked by subscripts, for example, by α k represents the face width-to-length ratio corresponding to the k-th two-dimensional frontal face photo, and is expressed by β k Represents the face area ratio corresponding to the k-th two-dimensional frontal face photo.
[0095] In step S304, different genders and different face shapes are distinguished, and the average estimated value of the width-to-length ratio of the j-th face shape of the i-th gender is calculated by the following formula:
[0096]
[0097] The average estimated value of the face area ratio of the jth face type of the i-th gender is calculated by the following formula
[0098]
[0099] Wherein k represents the number of two-dimensional photos of the face type of the gender, and the value of k in this embodiment is 100.
[0100] In this embodiment, when executing step S5, that is, determining the fitting weight according to the feature points corresponding to the face image to be processed and the feature points corresponding to the face base data, the following steps are specifically performed:
[0101] S501. Obtaining the feature point matrix M corresponding to the face image to be processed 2d ;
[0102] S502. Based on the feature point matrix M corresponding to the face image to be processed 2d The feature point matrix M corresponding to the face base data 3d , perform face pose estimation on the face image to be processed and obtain the rotation matrix R and displacement vector T;
[0103] S503. According to the formula
[0104]
[0105]
[0106] S504. Determine matrix A and matrix b; wherein, is the average face data corresponding to all face base data;
[0107] S505. According to formula w j+1 =w j +2·(Aw-b)A j Perform iterations to calculate the fitting weight w j ; where w={w1,w2,…,w m},
[0108] In this embodiment, according to the input two-dimensional face image to be processed, it is necessary to extract the feature points of the face. Common face analysis is mainly used for face or expression recognition, focusing on features such as eyebrows, eyes, nose, mouth, chin, etc., but lacks consideration of forehead feature points. Therefore, combined with the needs of building a three-dimensional face surface model and designing a personalized mask, the design Figure 3 The facial feature points shown, Figure 3 It includes the entire contour of the face, features of the cheeks, chin and other regions, as well as feature points of the eyebrows, eyes, nose, mouth and other regions. Preferably, 79 feature points are designed, including 10 feature points of eyebrows, 12 feature points of eyes, 9 feature points of nose, 20 feature points of mouth, 11 feature points of forehead, and 17 feature points of cheeks and chin. Considering that the accuracy of the automatic extraction algorithm of feature points is not high, the pure manual extraction method is time-consuming and the position accuracy is not high, so the Canny edge detection algorithm can be used to locate the feature edge and then extract the feature points manually. Therefore, when executing step S501, the following process can be executed:
[0109] 1. First, convert the two-dimensional face image to be processed into a grayscale photo, and then process each pixel p of the image;
[0110] 2. Perform Gaussian smoothing filtering on pixel p;
[0111] 3. Use the Sobel operator to calculate the first-order difference G of pixel p in the horizontal direction x ;
[0112] 4. Use the Sobel operator to calculate the first-order difference G of the pixel point p in the vertical direction y ;
[0113] 5. Calculate the grayscale gradient amplitude of pixel p
[0114] 6. Calculate the gray value gradient azimuth angle θ of pixel p = tan -1 (G y / G x );
[0115] 7. At each point, the pixel point p is compared with the two pixels along its corresponding gradient direction. If the pixel point is the maximum value, it is retained, otherwise the point is set to 0. This can suppress non-maximum values and retain the points with the largest local gradient to obtain refined edges.
[0116] 8. After the above steps are performed on each pixel of the two-dimensional photo, the edge where the grayscale features of the face change is obtained. In order to remove the edges with subtle changes, a dual threshold algorithm is used to detect and connect the edges. Preferably, the high threshold is set to 0.3 and the low threshold is set to 0.05. (1) Points less than the low threshold are discarded and assigned 0; points greater than the high threshold are immediately marked as edge positions; (2) Points less than the high threshold but greater than the low threshold are determined through 8-connected regions. If there is a marked point, it is marked as an edge position, otherwise the point is discarded.
[0117] 9. Based on the edge positions determined in the above steps, the feature points of the face are selected in sequence by using trained personnel interaction.
[0118] Through the above process 1-9, Figure 3 As shown, the feature points of the face are marked from the two-dimensional face image to be processed, and these feature points form a matrix M 2d .
[0119] In this embodiment, the principles of steps S502-S505 are:
[0120] After detecting a preset number of feature points M in the face image to be processed 2d After calibrating the preset number of feature points at the corresponding positions in the three-dimensional model, M3d , estimate the face pose. First, the similarity between the face image to be processed and the six different face shapes of the same gender i can be calculated by the following formula (7):
[0121]
[0122] Calculate the initial 3D model of the face:
[0123]
[0124] in and Calculated by equations (2) and (3) respectively, i indicates that the gender of the face is type i, j indicates one of the six types: Tian face, You face, Shen face, Guo face, Jia face, and Mu face, so the value of n′ is 6.
[0125] Considering that the face in the photo is often not in the center of the photo and the face may be tilted at a certain angle, it is necessary to estimate the face posture. According to the scaled orthographic projection model, the projection model of a three-dimensional face on a two-dimensional photo can be simulated. Assume that a vertex (x, y, z) in the three-dimensional space is T The pixel coordinates projected onto the two-dimensional plane are (u, v) T ,but:
[0126]
[0127] in is a rotation matrix that describes the facial orientation of the face; is a displacement vector that describes the offset of the face relative to the center of the image; s is a scalar scaling factor that describes the overall size of the face; It is a positive projection matrix that projects a three-dimensional space point onto a two-dimensional plane. Its form is as follows:
[0128]
[0129] For a preset number of feature points of a face, formula (1) can be extended to k dimensions, where k is the preset number of feature points, and the corresponding relationship between the three-dimensional face feature points and the face photo feature points can be obtained:
[0130]
[0131] The least squares method can be used to minimize the formula Get the estimated value of face pose and translation vector, where M 3dis a matrix composed of feature points of a 3D face model. The 3D face model used here is calculated by formula (8). 2d is the feature point matrix of the face in the photo, and f in the matrix P is x and f y It can be determined based on the pixel length and width of the photo, c x and c y is the center coordinate of the image.
[0132] By solving By minimizing R and T, we can obtain the rotation matrix R and displacement vector T.
[0133] In this embodiment, when executing step S6, that is, fitting each face base data according to the fitting weight to obtain the face 3D reconstruction model corresponding to the face image to be processed, the formula used is:
[0134]
[0135] Among them, S is the 3D reconstruction model of the face corresponding to the face image to be processed, w j is the fitting weight corresponding to the facial base data of the jth face type, m is the total number of face types corresponding to the facial base data, is the average face data corresponding to the face base data of the i-th gender and the j-th face type, is the average face data corresponding to the face base data of the i-th gender.
[0136] In this embodiment, the principle of executing step S6 is:
[0137] After obtaining the facial feature points and estimating the facial posture, the face is fitted using a 3D deformable model. The 3D reconstruction model of the face corresponding to the face image to be processed can be described by a vector S: S = (x1, y1, z1, ..., x n ,y n , z n ) T , where n is the number of vertices of the face mesh. Assuming that a new 3D face model is a linear combination of m other 3D faces, the face model fitted based on the photo can be described as:
[0138]
[0139] in is the average 3D face model of gender i in the face database, is the average face data corresponding to the i-th gender and the j-th face shape, w j represents the weight of the j-th face model, and its constraint satisfies 0≤w j ≤1, and its initial value is calculated by formula (7)ij Correspondingly, we can get Set the fitting weight w j The initial value of , i represents the subscript of the same gender.
[0140] In order to make the fitted 3D reconstruction model of the face as close as possible to the face and features in the face image to be processed, the formula Solve w = {w1, w2, ..., w m}, where R and T are the rotation matrix and translation vector respectively.
[0141] In order to pass Solve w = {w1, w2, ..., w m}, you can make The above formula can be written as follows:
[0142]
[0143] Suppose there is a function expression about weight w:
[0144] f(w)=(Aw-b) 2 (13)
[0145] According to the iterative formula of gradient descent Where γ is the learning rate of gradient descent, preferably γ = 0.001. Therefore, the iterative algorithm w of gradient descent can be obtained j+1 =w j +2·(Aw-b)A j .
[0146] In this embodiment, the formula w j+1 =w j +2·(Aw-b)A j Perform iterative calculations. For example, the number of iterations of the algorithm is set to 200, and the final obtained w j+1 It can be used as the fitting weight w required to execute step S5 j The obtained fitting weight w j The formula can be used to execute step S6 Calculation.
[0147] A computer program that executes the method for reconstructing the three-dimensional face model in this embodiment can be written and written into a computer device or storage medium. When the computer program is read out and run, the method for reconstructing the three-dimensional face model in this embodiment is executed, thereby achieving the same technical effect as the method for reconstructing the three-dimensional face model in the embodiment.
[0148] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, etc. used in the present disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "" and "the" used in the present disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by technicians in this technical field. The terms used in the specification of this embodiment are only for describing specific embodiments and are not intended to limit the present embodiment. The term "and / or" used in this embodiment includes any combination of one or more related listed items.
[0149] It should be understood that, although the term first, second, third etc. may be adopted in the present disclosure to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate the embodiment in the present embodiment, and unless otherwise required, the scope in the present embodiment will not be limited.
[0150] It should be appreciated that the embodiments in this embodiment can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method can be implemented in a computer program using standard programming techniques-including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner-according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed dedicated integrated circuit for this purpose.
[0151] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.
[0152] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or communicates with a charged particle tool or other imaging device, etc. Various aspects of this embodiment can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether it is removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques in this embodiment, the computer itself is also included in this embodiment.
[0153] The computer program can be applied to input data to perform the functions of the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present embodiment, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.
[0154] The above are only preferred embodiments of this embodiment. This embodiment is not limited to the above implementation. As long as the technical effect of this embodiment is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this embodiment should be included in the scope of protection of this embodiment. Within the scope of protection of this embodiment, its technical solution and / or implementation method can have various modifications and changes.
Claims
1. A method for reconstructing a three-dimensional face model, characterized in that: The method for reconstructing a three-dimensional face model comprises: Acquire multiple facial base data; Classifying each of the facial base data according to the gender and face shape of the person from whom it was collected, so that each of the facial base data is classified into a corresponding facial category; Calculate the average face data, the average estimated value of the width-to-length ratio and the average estimated value of the face shape area ratio corresponding to each of the face categories according to the face base data; Obtain the face image to be processed; Determining fitting weights according to feature points corresponding to the face image to be processed and feature points corresponding to the face base data; Fitting each of the face base data according to the fitting weights to obtain a face three-dimensional reconstruction model corresponding to the face image to be processed; According to the fitting weight, each of the face base data is fitted to obtain a face 3D reconstruction model corresponding to the face image to be processed, and the formula used is: in, is a 3D reconstruction model of a face corresponding to the face image to be processed, For the the fitting weights corresponding to the facial base data of the face shapes, is the total number of face base data corresponding to each gender, For the Gender, The average face data corresponding to the face base data of the face shape, For the Average face data corresponding to the face base data of the gender; The step of determining the fitting weights according to the feature points corresponding to the face image to be processed and the feature points corresponding to the face base data includes: Get the feature point matrix corresponding to the face image to be processed ; According to the feature point matrix corresponding to the face image to be processed The feature point matrix corresponding to the face base data , perform face pose estimation on the face image to be processed to obtain a rotation matrix and displacement vector ; According to the formula Determine the matrix and matrix ;in, is the average face data corresponding to all the face base data; According to the formula Perform iterations to calculate the fitting weights ;in, , .
2. The method for reconstructing a three-dimensional face model according to claim 1, characterized in that: The step of obtaining a plurality of face base data includes: Select multiple collection source personnel; each of the collection source personnel corresponds to a corresponding gender and face shape; the face shape is any one of the following: Tian face, You face, Shen face, Guo face, Jia face, and Mu face; Under a constant environment, use a three-dimensional laser scanning device to scan each of the acquisition source personnel to obtain a three-dimensional facial surface model; the three-dimensional facial surface model includes a plurality of vertices; The facial three-dimensional surface model is normalized, and dense correspondence is established between vertices in the facial three-dimensional surface model and the face of the acquisition source person to obtain the facial base data.
3. The method for reconstructing a three-dimensional face model according to claim 2, characterized in that: The step of calculating the average face data, the average estimated value of the width-to-length ratio and the average estimated value of the face area ratio corresponding to each of the face categories according to the face base data comprises: Performing cylindrical coordinate transformation on the face base data to obtain a two-dimensional frontal face photo; According to the formula and , calculate the face width-to-length ratio corresponding to the two-dimensional frontal face photo Compared with the face area ,in, and are respectively the width and height of the rectangular area where the face is located in the two-dimensional frontal photo of the face, is the area of the face region in the two-dimensional frontal face photo; Get All The corresponding Gender, The width-to-length ratio of the face shape corresponding to the two-dimensional frontal photo of the face shape is calculated according to the formula Calculate the average estimate of the aspect ratio ,in Show All In the two-dimensional frontal photograph of the face, The width-to-length ratio of the face shape corresponding to the two-dimensional frontal photograph of the face; Get All The corresponding Gender, The face area ratio corresponding to the two-dimensional frontal face photo of the face shape is calculated according to the formula Calculate the average estimated value of the face area ratio ,in Show All In the two-dimensional frontal photograph of the face, The face area ratio corresponding to the two-dimensional frontal face photograph.
4. The method for reconstructing a three-dimensional face model according to claim 1, characterized in that: The step of obtaining a feature point matrix corresponding to the face image to be processed ,include: Using the Canny edge detection algorithm, the feature edge of the face image to be processed is located to determine the edge position in the face image to be processed; Select feature points from the edge positions to form a matrix .
5. The method for reconstructing a three-dimensional face model according to claim 1, characterized in that: The feature point matrix corresponding to the face image to be processed The feature point matrix corresponding to the face base data , perform face pose estimation on the face image to be processed to obtain a rotation matrix and displacement vector ,include: By using the least squares method to solve the formula Minimize and determine the rotation matrix and the displacement vector ; in, , is the length of the face image to be processed, is the width of the face image to be processed, and is the center coordinate of the face image to be processed.
6. The method for reconstructing a three-dimensional face model according to claim 1, characterized in that: The iteration is performed according to the formula to calculate the fitting weight ,include: Obtain the face width-to-length ratio of the face image to be processed Compared with the face area ; According to the formula Set the fitting weights The initial value of is the average estimated value of the width-to-length ratio of all the face categories, is an average estimated value of the face area ratio for all the various face categories; According to the set initial value, according to the formula Perform gradient descent iterations to calculate the fitting weights .
7. A computer device, characterized in that: It comprises a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the method for reconstructing a three-dimensional face model as described in any one of claims 1-6.
8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to execute the method for reconstructing a three-dimensional face model as described in any one of claims 1 to 6 when executed by the processor.
Citation Information
Patent Citations
Single photograph-based fast face modeling method
CN107316340A
3D face reconstruction method based on convolutional neural network model
CN107680158A
Face image real-time beautifying and texture synthesizing method
CN110197462A
Face reconstruction method, system, platform and storage medium
CN110400369A
Three-dimensional human head and face model reconstruction method based on random face image
CN110443885A
