Method, system and device for automatic modeling of three-dimensional human faces based on single-view photos

Abstract

The present invention relates to the field of three-dimensional modeling and discloses a method, system and device for automatic three-dimensional face modeling based on a single-view photograph. The key points of the technical solution are as follows: detecting key points of the face in a frontal portrait photograph; selecting an underlying template model according to the frontal portrait photograph, selecting key point vertices corresponding to the key points of the face from the model vertices of the underlying template model, and performing a two-dimensional projection of the key point vertices on the plane where the frontal portrait photograph is located; fitting and generating a preliminary three-dimensional model according to the distance between the key points of the face and the two-dimensional projection positions of the key point vertices; projecting the model vertices of the preliminary three-dimensional model on a two-dimensional plane to obtain texture coordinates of the model vertices; preprocessing the frontal portrait photograph, and mapping the preprocessed frontal portrait photograph to the preliminary three-dimensional model according to the texture coordinates of the model vertices to obtain a target three-dimensional model, and a real, natural and complete target three-dimensional model can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional modeling, and more particularly to a method, system and device for automatic three-dimensional face modeling based on single-view photos. Background Art

[0002] 3D facial modeling is widely used in film, television, animation, and virtual reality. However, using 3D modeling software is expensive and inadequate for some applications. Acquiring 3D information from portrait photos typically requires multiple viewpoints, while using a single frontal photo simplifies the acquisition of material and makes facial modeling easier. Therefore, using a single frontal photo for facial modeling is more suitable for market needs.

[0003] However, using a single frontal photo for face modeling will result in the resulting model being unrealistic and incomplete. This is due to the lack of side face texture and contour features in a single frontal photo, as well as the quality of the photo acquisition. Therefore, these problems need to be solved. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, system and device for automatic modeling of three-dimensional human faces based on single-view photos, which can obtain a real, natural and complete three-dimensional model of the target.

[0005] The above technical objectives of the present invention are achieved through the following technical solutions: A method for automatically modeling a three-dimensional face based on a single-view photograph, comprising the following steps:

[0006] Detect the key points of the face in the frontal face photo;

[0007] Selecting an underlying template model based on the frontal face photo, selecting key point vertices corresponding to key points of the face from the model vertices of the underlying template model, and performing a two-dimensional projection of the key point vertices on the plane where the frontal face photo is located;

[0008] According to the distance between the facial key points and the two-dimensional projection positions of the key point vertices, a preliminary three-dimensional model is generated by fitting;

[0009] Project the model vertices of the preliminary three-dimensional model on a two-dimensional plane to obtain the texture coordinates of the model vertices;

[0010] The frontal portrait photo is preprocessed, and the preprocessed frontal portrait photo is mapped to the preliminary three-dimensional model according to the texture coordinates of the model vertices to obtain the target three-dimensional model.

[0011] As a preferred technical solution of the present invention, a method for detecting key facial points in a frontal portrait photo is: using a face detection algorithm based on a deep learning model to detect the frontal portrait photo and identify key facial points.

[0012] As a preferred technical solution of the present invention, the selecting of the underlying template model based on the frontal face portrait photo includes: selecting the underlying template model based on the stereoscopic degree of the face in the frontal face portrait photo.

[0013] As a preferred technical solution of the present invention, the step of mapping the pre-processed frontal portrait photo to the preliminary three-dimensional model according to the texture coordinates of the model vertices further includes the following sub-steps:

[0014] Adjust the positions of facial key points in shadowed frontal portraits;

[0015] By minimizing the distance between the adjusted facial key points and the two-dimensional projection positions of the key point vertices, a textured three-dimensional model is generated by fitting;

[0016] Obtaining texture coordinates of the adjusted model vertices according to the textured three-dimensional model;

[0017] Obtain the texture map corresponding to the key points of the face according to the texture coordinates of the adjusted model vertices;

[0018] The obtained map is applied to the preliminary 3D model.

[0019] As a preferred technical solution of the present invention, the method of fitting and generating a preliminary three-dimensional model based on the distance between the two-dimensional projection positions of the facial key points and the key point vertices includes: fitting and generating a preliminary three-dimensional model by minimizing the distance between the two-dimensional projection positions of the facial key points and the key point vertices.

[0020] As a preferred technical solution of the present invention, after obtaining the target three-dimensional model, the map on the target three-dimensional model is repaired and texture supplemented to obtain an optimized three-dimensional model;

[0021] Map repair of the target 3D model includes the following sub-steps:

[0022] Find the area to be repaired based on the 2D projection of the target 3D model;

[0023] Generate a sketch of the area to be repaired based on the boundary calculation of the area to be repaired;

[0024] The area to be repaired is obtained through face segmentation technology as the repair mask;

[0025] Input the sketch and mask into the generative model of image face re-editing to regenerate the area to be repaired;

[0026] Texture supplementation of the target 3D model includes the following sub-steps:

[0027] Finding a portion of the target three-dimensional model where texture is missing, and replacing the portion with a preset texture corresponding to the portion where texture is missing;

[0028] According to the face area of ​​the portrait photo, the average skin color is obtained and all skin colors of the target 3D model are filled and replaced according to the average skin color.

[0029] Perform feathering fusion processing on the skin color at the boundary of the target 3D model.

[0030] As a preferred technical solution of the present invention, the projection is a perspective projection.

[0031] As a preferred technical solution of the present invention, the process of preprocessing the front-facing portrait photo is: using Gaussian filtering to perform image denoising on the front-facing portrait photo, using a low-light image enhancement method to adjust the light and dark contrast of the front-facing portrait photo, and removing shadows in the front-facing portrait photo.

[0032] A 3D face automatic modeling system based on single-view photos and texture restoration, comprising:

[0033] Key point detection module: detects key points of the face in the frontal face photo;

[0034] Model and vertex selection module: used to select the underlying template model based on the front face photo, select the key point vertices corresponding to the key points of the face from the model vertices of the underlying template model, and perform two-dimensional projection of the key point vertices on the plane where the front face photo is located;

[0035] Fitting module: used to fit and generate a preliminary 3D model based on the distance between the facial key points and the 2D projection positions of the key point vertices;

[0036] Texture coordinate calculation module: used to project the model vertices of the preliminary three-dimensional model on a two-dimensional plane to obtain the texture coordinates of the model vertices;

[0037] Preprocessing and mapping module: used to preprocess the portrait front face photo, and map the preprocessed portrait front face photo to the preliminary three-dimensional model according to the texture coordinates of the model vertices to obtain the target three-dimensional model.

[0038] A device for automatic three-dimensional face modeling based on single-view photos and texture restoration is provided. The device is equipped with the system described above and executes the method described above.

[0039] In summary, the present invention has the following beneficial effects: it can use an underlying template model with any number of vertices to fit a frontal portrait photo to obtain a preliminary three-dimensional model, so that during actual fitting, the selection range of the underlying template model is wider, which is more conducive to practical use;

[0040] When fitting a frontal portrait photo, the underlying template model is perspective projected to make the vertex projection coordinates more closely match the actual photo, making the shape and surface condition function of the preliminary 3D model more accurate, and also making the texture coordinates more precise, resulting in better texture mapping effects.

[0041] Before mapping, pre-process the frontal face photo to remove shadows and obtain normal texture, so that the mapping will be more realistic.

[0042] When mapping, we also re-fit specific areas to obtain more accurate regional textures, making the mapping more realistic.

[0043] After mapping, the preliminary 3D model will be repaired and supplemented to make the missing textures complete and the unnatural texture parts uniformly adjusted, so as to obtain a more realistic, natural and complete target 3D model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the method of the present invention;

[0045] Figure 2 Schematic diagram of key points of a face according to the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below with reference to the accompanying drawings.

[0047] The present invention provides a method for automatically modeling a three-dimensional face based on a single-view photograph. Figure 1 As shown, the following steps are included:

[0048] S1. Use the face detection algorithm based on the deep learning model to detect the frontal face photo and identify the key points of the face. For example, you can choose to identify 68 key points of the face, such as Figure 2 As shown, and set K = [k1, k2, ...k 68 ], these 68 points represent the key points of the face contour, eyebrows, eyes, nose, and mouth. The face detection algorithm includes a face monitoring model and a key point prediction model. When used, a frontal portrait photo is input into the face detection model to obtain a face detection frame. The face detection frame is then used to crop the entire frontal portrait photo to obtain the face region. Finally, the face region is input into the key point prediction model to obtain the corresponding face key points.

[0049] S2. Select an underlying template model based on the frontal face photo. For example, a flame model can be selected as the underlying template model. The FLAME model can be a model with any vertex distribution and any number of vertices. The underlying template model can be set to T, and the number of its model vertices is n.

[0050] Assume the shape deformation function is: in represents the shape factor, Represents different shape deformations. Shape deformation refers to the deformation of facial features based on the standard face model, such as different face shape deformations such as long face, round face, square face, and pointed chin, eye deformations such as big eyes and slender eyes, and lip deformations such as thick lips and thin lips.

[0051] Assume that the case variable function is in represents the expression shape coefficient, It represents different facial deformations. The facial deformation refers to the deformation of facial features based on the expressionless face model, such as upward or downward corners of the mouth, frowning, and raising eyebrows.

[0052] Then the underlying template model can be expressed as follows after deformation: The fitting process in the latter step is actually to obtain the corresponding shape deformation function and surface deformation function, so as to adjust the underlying template model and obtain a preliminary three-dimensional model.

[0053] S3, select the key point vertices corresponding to the key points of the face on the underlying template model, and perform a two-dimensional perspective projection on the plane where the portrait face photo is located; select the model The key point vertices on the face correspond to the 68 key points on the face Calculate the perspective projection of V on a two-dimensional plane For n=1,...68, in

[0054] f is the focal length of perspective projection (successive projection), R is the rotation matrix, and C is the translation vector.

[0055] S4, by minimizing the distance between the two-dimensional projection coordinates of the face key points and the key point vertices, a preliminary three-dimensional model is generated by fitting; the objective function of the fitting model is the projection V of the key point vertices p The distance from the facial key point K Minimize the distance between the projection of the key point vertex and the face key point and express it as Then the deformation parameters are obtained Thus, a preliminary three-dimensional model generated by fitting is obtained

[0056] In this step, an underlying template model with any number of vertices can be used to fit the frontal portrait photo to obtain a preliminary three-dimensional model, so that during actual fitting, the selection range of the underlying template model is wider and more conducive to practical use.

[0057] S5. Project the model vertices of the preliminary three-dimensional model on a two-dimensional plane to obtain the texture coordinates of the model vertices; calculate the model Each model vertex m i Projection on a two-dimensional plane Then the texture coordinates of each model vertex can be obtained as

[0058] S6. Preprocessing the frontal portrait photo. The process is as follows: The restoration process of the frontal portrait photo is as follows: using Gaussian filtering to perform image denoising on the frontal portrait photo, using a low-light image enhancement method to adjust the light and dark contrast of the frontal portrait photo, and removing shadows in the frontal portrait photo. The low-light image enhancement method uses photos under insufficient lighting and normal lighting as training data to train a neural network model, thereby achieving photo lighting adjustment and restoring the heavily shadowed image to the appearance under sufficient lighting. In the present invention, the abnormal texture caused by shadows in the frontal portrait photo can be converted into normal texture through the above processing.

[0059] S7, mapping the pre-processed frontal face photo of the person to the preliminary three-dimensional model according to the texture coordinates of the model vertices to obtain a target three-dimensional model; specifically, the step of adjusting the positions of key facial points of the person with shadows in the frontal face photo of the person;

[0060] By minimizing the distance between the adjusted facial key points and the two-dimensional projection positions of the key point vertices, a textured three-dimensional model is generated by fitting;

[0061] Obtaining texture coordinates of adjusted key point vertices according to the texture three-dimensional model;

[0062] Obtain the texture map corresponding to the key points of the face according to the texture coordinates of the adjusted key point vertices;

[0063] The obtained map is applied to the preliminary 3D model.

[0064] For example, if you use the texture coordinates fitted with key points directly for mapping, the lip texture will usually retain the shadows between the upper and lower lips and the corners of the mouth. Therefore, in order to obtain the shadows of the mouth, you can only adjust the 6 key points of the lips, such as Figure 2 As shown, the three key points on the edge of the upper lip (points 66, 67, 68) are adjusted upward, and the three key points on the edge of the lower lip (points 62, 63, 64) are adjusted downward to obtain a new set of facial key points K M =[k1, k2, ...k 68], then minimize the distance between the new facial key points and the 2D projections of the key point vertices on the underlying template model, fitting the textured 3D model. This then yields the adjusted texture coordinates of the key point vertices corresponding to the mouth position. Using these texture coordinates, a normal mouth texture can be obtained. Once the normal mouth texture is obtained, it can be mapped onto the texture coordinates corresponding to the preliminary 3D model, completing the mouth texture mapping.

[0065] Similarly, the cheek fitting will also cause part of the cheek texture coordinates to fall on the image background or hair, resulting in errors in the cheek mapping. In a similar way, adjust the facial key points (points 1 to 17) to shrink, obtain a new set of facial key points, and then perform fitting, that is, minimize the distance between the projection of the model key point vertex and the facial key point Calculate the texture coordinates to obtain the normal cheek texture, and then map the normal cheek texture onto the preliminary 3D model to complete the texture mapping of the cheek.

[0066] S8. As a further optimization of the target three-dimensional model, the target three-dimensional model is also subjected to mapping repair and texture supplementation to obtain an optimized three-dimensional model.

[0067] The mapping repair of the target 3D model includes:

[0068] Map repair of the target 3D model includes the following sub-steps:

[0069] Find the area to be repaired based on the 2D projection of the target 3D model;

[0070] Generate a sketch of the area to be repaired based on the boundary calculation of the area to be repaired;

[0071] The area to be repaired is obtained through face segmentation technology as the repair mask;

[0072] Input the sketch and mask into the generative model of image face re-editing to regenerate the area to be repaired.

[0073] For example: Hair modeling is independent of the head. In order to eliminate the effect of hair covering the forehead in the photo, that is, the hair texture, the hairstyle map needs to be converted to a bald head. Use SC-FEGAN to repair the forehead texture and replace the hair texture with skin color to get a bald head shape. SC-FEGAN is a generative model that realizes face re-editing in pictures. It regenerates the mask area of ​​the face by adding sketches and mask information to the face picture. Given a frontal face picture of a person, set the hair area as a mask, and re-add the hair sketch along the forehead. SC-FEGAN will generate the same texture as the sketch. Figure 1The part of the original image covered by the hair is generated to connect with the natural forehead skin, thereby achieving the purpose of repairing the forehead texture.

[0074] The above solution can also be used to perform texture restoration on other boundary positions of the face, thereby effectively avoiding the problem of unnatural connection between map textures.

[0075] Texture supplementation of the target 3D model includes the following sub-steps:

[0076] Finding a portion of the target three-dimensional model where texture is missing, and replacing the portion with a preset texture corresponding to the portion where texture is missing;

[0077] According to the face area of ​​the portrait photo, the average skin color is obtained and all skin colors of the target 3D model are filled and replaced according to the average skin color.

[0078] Perform feathering fusion processing on the skin color at the boundary of the target 3D model.

[0079] For example, supplement the ears, neck, top of the head, back of the head, etc.;

[0080] Specifically, the ear texture is supplemented. Since the ear has no key points, the ear texture is difficult to obtain by fitting the underlying template model to the front face photo of the portrait, so the ear part can be replaced with a fixed ear texture. The fixed ear texture can be selected from the modeling database or obtained by actual modeling.

[0081] When filling the neck, top of the head, and back of the head with texture, the average RGB value of the skin color in the face area is extracted based on individual differences in skin color, expressed as Fill in, where [r i , g i , b i ], i∈1, ..., m represents the RGB value of the i-th pixel. The boundary is fused using feathering operation to make the skin color of the whole head blend naturally. The pixel value of the original image position (x, y) is represented by vector Indicates that the pixel value of the target image to be replaced is expressed as The feathering step can be expressed as Where dist is the minimum distance from the point (x, y) to the boundary.

[0082] By supplementing the texture of the mapped target three-dimensional model as described above, a complete optimized three-dimensional model with relatively uniform head skin can be obtained.

[0083] It should be noted that when there is only a frontal portrait photo, the underlying template model can only fit the facial key points in the two-dimensional plane, and cannot fit the three-dimensionality of the face. Therefore, when selecting the underlying template model, you can choose according to the three-dimensionality of the face in the frontal portrait photo, so that it can be closer to the real face in terms of three-dimensionality.

[0084] The projections in the present invention are all perspective projections, which can be more consistent with the projection of the actual camera, so that the projection coordinates of the key point vertices and the model vertices are more in line with the actual photos, making the shape and surface condition function of the preliminary three-dimensional model more accurate. It can also avoid the problem of texture deformation caused by the use of orthogonal projection when fitting pictures with the traditional flame model, so that the texture mapping effect is better.

[0085] Corresponding to the above-mentioned method of the present invention, a 3D face automatic modeling system based on single-view photos and texture restoration is also provided, comprising:

[0086] Key point detection module: detects key points of the face in the frontal face photo;

[0087] Model and vertex selection module: used to select the underlying template model based on the front face photo, select the key point vertices corresponding to the key points of the face from the model vertices of the underlying template model, and perform two-dimensional projection of the key point vertices on the plane where the front face photo is located;

[0088] Fitting module: used to fit and generate a preliminary 3D model based on the distance between the facial key points and the 2D projection positions of the key point vertices;

[0089] Texture coordinate calculation module: used to project the model vertices of the preliminary three-dimensional model on a two-dimensional plane to obtain the texture coordinates of the model vertices;

[0090] Preprocessing and mapping module: used to preprocess the portrait front face photo, and map the preprocessed portrait front face photo to the preliminary three-dimensional model according to the texture coordinates of the model vertices to obtain the target three-dimensional model.

[0091] The present invention also provides a device for automatic three-dimensional face modeling based on single-view photos and texture restoration, which is installed with the system as described above and executes the method as described above.

[0092] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for automatic modeling of three-dimensional human faces based on single-view photos, characterized by: The steps include: Detect the key points of the face in the frontal face photo; Selecting an underlying template model based on the frontal face photo, selecting key point vertices corresponding to key points of the face from the model vertices of the underlying template model, and performing a two-dimensional projection of the key point vertices on the plane where the frontal face photo is located; According to the distance between the facial key points and the two-dimensional projection positions of the key point vertices, a preliminary three-dimensional model is generated by fitting; Project the model vertices of the preliminary three-dimensional model on a two-dimensional plane to obtain the texture coordinates of the model vertices; Preprocessing the frontal face photo of the portrait, and mapping the preprocessed frontal face photo of the portrait to the preliminary three-dimensional model according to the texture coordinates of the model vertices, including: adjusting the positions of key facial points with shadows in the frontal face photo of the portrait; A textured three-dimensional model is generated by fitting by minimizing the distance between the adjusted facial key points and the two-dimensional projection positions of the key point vertices; the texture coordinates of the adjusted model vertices are obtained based on the textured three-dimensional model; a map corresponding to the facial key points is obtained based on the texture coordinates of the adjusted model vertices; the obtained map is attached to the preliminary three-dimensional model; and a target three-dimensional model is obtained.

2. The method for automatic 3D face modeling based on single-view photos according to claim 1, characterized in that: The method for detecting the key points of the face in the frontal portrait photo is: using a face detection algorithm based on a deep learning model to detect the frontal portrait photo and identify the key points of the face.

3. The method for automatic 3D face modeling based on single-view photos according to claim 1, characterized in that: The selecting of the underlying template model according to the frontal face portrait photo includes: selecting the underlying template model according to the three-dimensionality of the face in the frontal face portrait photo.

4. The method for automatic 3D face modeling based on single-view photos according to claim 1, wherein: The method of fitting and generating a preliminary three-dimensional model according to the distance between the two-dimensional projection positions of the key points of the face and the key point vertices includes: fitting and generating the preliminary three-dimensional model by minimizing the distance between the two-dimensional projection positions of the key points of the face and the key point vertices.

5. The method for automatic 3D face modeling based on single-view photos according to claim 1, wherein: After obtaining the target 3D model, repairing and texture supplementing the map on the target 3D model are performed to obtain an optimized 3D model; Map repair of the target 3D model includes the following sub-steps: Find the area to be repaired based on the 2D projection of the target 3D model; Generate a sketch of the area to be repaired based on the boundary calculation of the area to be repaired; The area to be repaired is obtained through face segmentation technology as the repair mask; Input the sketch and mask into the generative model of image face re-editing to regenerate the area to be repaired; Texture supplementation of the target 3D model includes the following sub-steps: Finding a portion of the target three-dimensional model where texture is missing, and replacing the portion with a preset texture corresponding to the portion where texture is missing; According to the face area of ​​the portrait photo, the average skin color is obtained and all skin colors of the target 3D model are filled and replaced according to the average skin color. Perform feathering fusion processing on the skin color at the boundary of the target 3D model.

6. The method for automatic 3D face modeling based on single-view photos according to claim 5, characterized in that: The projection is a perspective projection.

7. The method for automatic 3D face modeling based on single-view photos according to claim 1, wherein: The process of preprocessing the front-facing portrait photos is as follows: using Gaussian filtering to perform image denoising on the front-facing portrait photos, using low-light image enhancement methods to adjust the light and dark contrast of the front-facing portrait photos, and removing shadows in the front-facing portrait photos.

8. A 3D face automatic modeling system based on single-view photos and texture restoration, characterized by: include: Key point detection module: detects key points of the face in the frontal face photo; Model and vertex selection module: used to select the underlying template model based on the front face photo, select the key point vertices corresponding to the key points of the face from the model vertices of the underlying template model, and perform two-dimensional projection of the key point vertices on the plane where the front face photo is located; Fitting module: used to fit and generate a preliminary 3D model based on the distance between the facial key points and the 2D projection positions of the key point vertices; Texture coordinate calculation module: used to project the model vertices of the preliminary three-dimensional model on a two-dimensional plane to obtain the texture coordinates of the model vertices; Preprocessing and mapping module: used to preprocess the frontal portrait photo and map the preprocessed frontal portrait photo to the preliminary 3D model according to the texture coordinates of the model vertices, including: adjusting the position of the key points of the face with shadows in the frontal portrait photo; A textured three-dimensional model is generated by fitting by minimizing the distance between the adjusted facial key points and the two-dimensional projection positions of the key point vertices; the texture coordinates of the adjusted model vertices are obtained based on the textured three-dimensional model; a map corresponding to the facial key points is obtained based on the texture coordinates of the adjusted model vertices; the obtained map is attached to the preliminary three-dimensional model; and a target three-dimensional model is obtained.

9. A device for automatic 3D face modeling based on single-view photos and texture restoration, characterized by: The device is installed with the system according to claim 8 and performs the method according to any one of claims 1 to 7.

Images