A method for automatically generating a three-dimensional face model
By loading user face images and standard face models, and utilizing semantic key point detection and retopology techniques, combined with 2D and 3D mapping, mesh deformation, and smoothing methods, a high-quality 3D face model conforming to a specified topological structure is generated. This solves the problem of fixed topological relationships in existing technologies and improves computational performance and flexibility.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LANZHOU FUTURE NEW FILM CULTURE & TECH GRP CO LTD
- Filing Date
- 2022-10-18
- Publication Date
- 2026-05-19
AI Technical Summary
Existing computer vision methods generate 3D face models with fixed topological relationships and a large number of mesh vertices, resulting in insufficient computational performance and flexibility, making it difficult to meet the needs of practical applications, especially when 3D applications require a whole human body or head model, and the model designer needs to specify the topological structure.
By loading user face images and standard face models, and utilizing semantic key point detection and retopology techniques, combined with 2D and 3D mapping, mesh deformation and smoothing methods, a user face model conforming to a specified topological structure is generated. Mesh distortion is optimized and the models are then fused together to generate a high-quality 3D face model.
It enables the generation of accurate 3D face models with precise face shapes and textures based on production needs, reducing post-processing work, improving the quality and integrity of the models, and meeting the computational performance and flexibility requirements of practical applications.
Smart Images
Figure CN115546409B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of motor technology, specifically to a method for automatically generating three-dimensional human face models. Background Technology
[0002] 3D face model generation is a process that involves collecting facial information and using image processing methods, computer vision principles, 3D mesh processing, and 3D point cloud processing to calculate the 3D information of the face. Based on the different data acquisition and generation principles, 3D face model generation technology can be divided into 2D image generation and 3D scanning acquisition. 2D image generation generally refers to reconstructing the 3D information of a face from an image; 3D scanning acquisition refers to capturing the 3D information of a face from a 3D scanning device and then performing post-processing. The 2D image generation method for 3D face models has the advantages of wide applicability, low cost, and high efficiency, and has a very broad application space in virtual reality, games, live streaming, and film.
[0003] Currently, the generation of 3D face models based on 2D images mainly relies on computer vision methods. These methods train a generative model using manually annotated face image datasets and parameterized deformable face models. The parameterized deformable face model allows for parameter adjustments to produce different face shapes and expressions. For a given face image, the generated 3D face model can project an image with minimal residuals to the original face image and maintains a consistent topological relationship with the parameterized deformable face model. The parameterized deformable face model, however, has a denser mesh with a relatively large data volume.
[0004] 3D face models obtained through computer vision methods have fixed topological relationships and a large number of mesh vertices, making them difficult to apply to applications with high requirements for computational performance and flexibility.
[0005] On the one hand, while 3D face models obtained through computer vision methods can recreate the texture and shape information of a face relatively well, their fixed topological relationships prevent them from being flexibly applied to practical systems. This is especially true for 3D applications, which typically require a complete human body model or a full head model, and the model designer must specify the model's topological structure. Therefore, 3D face models obtained through computer vision methods do not meet these requirements. On the other hand, 3D face models obtained through computer vision methods have dense mesh vertices, which negatively impacts computational performance in both application system rendering and model updates. Summary of the Invention
[0006] The purpose of this invention is to provide a method for automatically generating three-dimensional human face models to solve the problems mentioned in the background art.
[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for automatically generating a 3D face model, the specific steps of which are as follows:
[0008] Data loading and processing involves two types of data. The first is loading the user's face image I. usr Secondly, there is the standard face model;
[0009] User's face image I usr This refers to images that primarily depict the front of a person's face, and semantic key points are detected after the image is loaded.
[0010] A standard face model refers to a 3D model that includes a facial region or other human body parts besides the facial region. The facial region of a standard face model is defined during the design process, and the 3D facial information is M based on the specified facial region. std and texture information I std The texture information of the frontal region of the face is extracted and then projected onto the image to obtain the standard face model projection image I. std_prj Then, project the standard face model onto image I. std_prj Detecting semantic key points P key_std ;
[0011] User Model M Based on Computer Vision usr Generate semantic key points P from a single user image and combine them with the facial image. key_std Automatically generate user model M usr The user model M usr Contains user texture information I std And the user's facial shape information;
[0012] Using semantic key point P key_std Optimizing user model M with facial edge information usr Each semantic key point in a face image can be displayed in the user model M. usr Each 3D vertex corresponds to a fixed index in the user model; these 3D vertices are called the 3D keypoints V of the user model. key ;
[0013] The user model mesh retopology process is to retopologically remodel the user model M. usr The topological structure is reconstructed to form the topological structure of the standard face model. This process utilizes texture information I. std Using the model's 3D information, calculate the 2D and 3D mappings respectively to obtain the retopologically reconstructed user face model;
[0014] Boundary constraints are added during the mapping calculation process. The mesh vertices are connected to the mapping points, and the angle between the connecting line and the normal plane of the mapping point, the angle between the normal vectors of the two points, and the length of the connecting line are calculated. Thresholds are set for these three values. If they are outside the range, they are considered to be outside the boundary and no mapping is performed.
[0015] The local shape optimization process of the model mesh is to optimize the deformed mesh of the user face model after retopology based on the surface geometric information of the standard face model mesh. Deformation refers to the sharpness, splitting and overlapping of the mesh. For the vertex of the standard face mesh after retopology, the adjacent points of the point in the original standard face model mesh are rigidly registered to the space after retopology in order to find the correction point of the current point.
[0016] After mesh optimization is complete, the UV coordinates of the retopologically optimized user face model on the texture image need to be updated. For each optimized mesh vertex, the user model M needs to be found. usr Multiple nearest points are identified, and new UV coordinates are obtained by interpolating the UV coordinates of these nearest points and then assigned to the retopologically reconstructed user face model.
[0017] The model mesh fusion and stitching process involves fusing the retopologically reconstructed user face model with the standard face model, integrating the retopologically reconstructed model containing user face information into the standard face model to generate the user standard model.
[0018] Output the result model, which will output the user's standard model.
[0019] Furthermore, the standard face model includes skeletal and skin information, enabling it to simulate human movements and expressions.
[0020] Furthermore, the user's facial shape information includes a facial average model T, shape parameter alpha, shape basis S, expression parameter beta, and expression basis E, and a user model M. usr The expression for three-dimensional information is:
[0021] M usr =T + alpha*S + beta*E.
[0022] Furthermore, the 3D mapping utilizes rigid registration and mesh deformation to increase the overlap between the surfaces of the user's face model and the standard face model in space, thereby finding a mapping relationship to achieve the purpose of retopology. The direction of registration and deformation is from the standard face model to the user's face model. Rigid registration can ensure the maximum overlap between the two models in space without changing the shape of the model.
[0023] Furthermore, semantic key points P of facial images are utilized. key_std and user model 3D key points V key The projection point Vkey_proj The error constructs a problem-solving model to solve for the optimal pose R, shape parameter alpha, and expression parameter beta.
[0024] In the above formula, n represents the number of key points involved in the optimization. When the user model M usr If the initial pose shows a profile view that reaches a certain threshold, then the occluded key points will not participate in the optimization. Therefore, the value of n is based on the user model M. usr Initial attitude determined.
[0025] Furthermore, the 2D mapping utilizes the user model M usr Projected coordinates, user facial semantic key points P key Semantic key points P projected from the standard face model key_std By performing an affine transformation on the projected coordinates in two-dimensional space, the model from the standard model to the user model M can be found. usr The initial mapping.
[0026] Furthermore, both mesh deformation and mesh smoothing can achieve mesh fusion. Mesh smoothing is a smoothing operation for sharp protrusions or depressions that appear locally at the joints. Mesh deformation is a deformation operation on the model of the unretopologized parts by using the edge points of the retopologized parts as control points.
[0027] Compared with the prior art, the beneficial effects of the present invention are:
[0028] This solution can generate user face models according to a specified topology, flexibly providing accurate result models based on production needs. The user model generation process fully utilizes facial image data, resulting in more precise facial shapes and more accurate texture mapping. During the user model mesh retopology process, it fully leverages 2D texture and 3D mesh information, making mesh vertex mapping more robust. The retopology process combines mesh generation, mesh deformation, and mesh smoothing methods. The mesh generation method uses key points based on the face to ensure accurate 2D mapping; mesh deformation and smoothing methods, under reasonable deformation, make the user model and standard face model closer in 3D space, resulting in more accurate mapping; the retopology model mesh local shape optimization uses original mesh information to correct mesh distortion and improve model quality. The model mesh fusion and stitching uses mesh deformation and mesh smoothing methods, ensuring that the output meets design requirements while maintaining model integrity and high quality, reducing post-processing work. Attached Figure Description
[0029] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0032] Example:
[0033] Please see Figure 1 The present invention provides a technical solution: the process included in the method proposed in this case is characterized by:
[0034] a) Data loading and processing, mainly targeting two types of data. The first is loading the user's face image I. usr Secondly, there is the standard face model.
[0035] User face images refer to images with the front of the face as the main part. In this case, semantic key points will be detected after the image is loaded.
[0036] A standard face model refers to a 3D model that includes a face or other human body parts. Standard face models can include skeletal and skinning information, and can simulate human movements and expressions. The facial region of a standard face model is pre-defined during the design phase; this project uses the specified facial region from the standard face model to extract the 3D facial information. std and texture information I std Extract the texture information. Then project the texture information of the frontal region of the face onto the image to obtain the standard face model projection image I. std_prj Then, project the standard face model onto image I. std_prj Semantic key points P for face detection in China key_std .
[0037] b) User model M based on computer vision usr The generation process involves taking a single user image and combining it with the semantic key points P of the face image. key_std Automatically generate user model M usr The user model M usrIt contains the user's facial texture information and facial shape information. The shape information includes the average facial model T, shape parameter alpha, shape basis S, expression parameter beta, and expression basis E. The expression for the user model's 3D information is:
[0038] M usr =T + alpha*S + beta*E
[0039] Due to the directly generated user model M usr The projection onto the user's face image exhibits bias, primarily due to the pose and shape of the user model. This bias is addressed by optimizing the user model using semantic key points and facial edge information. Each semantic key point in the face image can be mapped onto the user model M. usr In the model, each point corresponds to a 3D vertex with a fixed index; these points are called the 3D keypoints V of the user model. key For user model M usr This case addresses the projection error problem on user facial images by utilizing semantic key points P in facial images. key and user model 3D key points V key The projection point V key_proj The error construct problem is solved by modeling the optimal pose R, shape parameter alpha, and expression parameter beta.
[0040]
[0041] In the above formula, n represents the number of key points involved in the optimization. When the user model's initial pose shows a side profile that reaches a certain threshold, the occluded key points will not participate in the optimization. Therefore, the value of n is determined based on the user model's initial pose.
[0042] c) The user model mesh retopology process reconstructs the topology of the user model into the topology of a standard face model. This process utilizes facial texture information and the model's 3D information to calculate 2D and 3D mappings respectively, resulting in a retopologically reconstructed user face model. 3D mapping can be used independently, but using 2D mapping alone may lead to mapping errors. Combining both methods yields better results.
[0043] 2D mapping utilizes the projected coordinates of the user model and the semantic key points P of the user's face. key Projecting semantic key points P of standard face model key_std This method involves performing an affine transformation on the projected coordinates in two-dimensional space to find a preliminary mapping from the standard model to the user model. This mapping method is computationally efficient and faster.
[0044] 3D mapping utilizes methods such as rigid registration and mesh deformation to maximize the overlap between the surfaces of a user's face model and a standard face model in space, thereby establishing a mapping relationship for retopology purposes. The registration and deformation directions are from the standard face model to the user's face model. Rigid registration ensures maximum overlap between the two models in space without altering their shapes. Because the shapes of the standard and user face models differ, using only rigid registration to find the mapping can lead to serious mapping errors in facial features. Mesh deformation involves altering the shape of the model to fit the target model, making the surfaces of the two models closer in space. However, mesh deformation works best when the two models are roughly registered; therefore, rigid registration must be performed first, followed by mesh deformation. Combining both methods ensures the stability of the mapping.
[0045] While 2D mapping can find mappings for most model vertices, it often fails to accurately represent occluded or densely overlapping projection areas, such as nostrils, nasal wings, and corners of the mouth. 3D mapping can also obtain mappings for most model vertices, but calculating the correspondence solely in 3D space ignores texture information and is computationally intensive. This design first performs 2D mapping to provide accurate basis for scale and rigidity transformations, then performs 3D mapping, ensuring both computational performance and system stability.
[0046] In this case, the mapping point calculation method mainly involves finding the nearest point in space with normal vector constraints. Normal vector constraints refer to the angle constraint between the normal vectors of two points; the larger the angle, the lower the matching degree. Using only the nearest point in space method will result in mapping misalignment in densely populated facial areas. Both methods are within the protection scope of this case.
[0047] Since the boundaries of the standard face model and the user model are likely to be inconsistent, this case incorporates boundary constraints during the mapping calculation process. Specifically, it connects the grid vertices to the mapping points, calculates the angle between the connecting line and the normal plane of the mapping point, the angle between the normal vectors of the two points, and the length of the connecting line, and sets thresholds for these three values. If the values are outside the range, they are considered to be outside the boundary and no mapping is performed.
[0048] d) The local shape optimization process of the model mesh involves optimizing the deformed mesh of the retopologically retopographed user face model based on the surface geometry information of the standard face model mesh. Deformation refers to phenomena such as sharp edges, splits, and overlaps in the mesh. In this case, for each vertex of the retopographed standard face mesh, its adjacent points in the original standard face model mesh are rigidly registered to the retopographed space to find the correction point for the current point.
[0049] After mesh optimization is complete, the UV coordinates of the retopologically optimized user face model on the texture image need to be updated. This approach involves finding multiple nearest points in the user model for each optimized mesh vertex, interpolating the UV coordinates of these nearest points to obtain new UV coordinates, and then assigning them to the retopologically optimized user face model.
[0050] e) The model mesh fusion and stitching process involves fusing the retopologically reconstructed user face model with the standard face model. This integrates the retopologically reconstructed model containing user facial information into the standard face model, generating the user standard model. This process first performs scale transformation and rigid registration of the retopologically reconstructed model to the initial standard face model. The transformation and registration direction is from the retopologically reconstructed model to the standard face model. Then, mesh deformation and smoothing operations are used to ensure the final user standard model is complete and smooth.
[0051] The mesh deformation method uses the edge points of the retopologized portion as control points to deform the unretopologized portion of the model, resulting in natural transitions at the seams and sides of the model surface. In this case, various deformation methods can be used, such as local rigidity-based deformation (as rigid as possible) or Laplace deformation.
[0052] Mesh smoothing is an operation that smooths out sharp protrusions or depressions at the seams, ensuring that the merged face model is more complete and aesthetically pleasing.
[0053] In this case, both mesh deformation and mesh smoothing can achieve the fusion function. Mesh deformation can make the transition of the curved surfaces on both sides of the fused joint natural, with minimal effect on small protrusions and depressions at the joint. Mesh smoothing alone can only solve the local smoothing of the joint, but the transition of the curved surfaces on both sides of the joint may not be natural. Therefore, this case combines the two to ensure the overall naturalness and local smoothness of the fused result.
[0054] f) Output the result model, and output the user's standard model. In this case, the user's standard model is saved to the path specified by the user, based on the format of the standard face model.
[0055] The entire process of the system proposed in this case, that is, the overall process of the system, is an innovation.
[0056] The user model generation process combines computer vision methods and facial semantic key points to generate high-quality user models.
[0057] User model mesh retopology combines facial texture information, model mesh geometry, and semantic key points from facial images, utilizing spatial transformations, mesh generation, and mesh deformation methods. The combination of 2D and 3D mapping results in more stable computational results. 3D mapping can be used alone, but 2D mapping alone is not feasible.
[0058] Local shape optimization of the model mesh is performed by using the original mesh information to locally optimize the retopology results.
[0059] The model mesh fusion and stitching method combines mesh deformation and mesh smoothing to make the resulting model more complete and of higher quality.
[0060] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or basic characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the invention is defined by the appended claims rather than the foregoing description. Therefore, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.
[0061] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for automatically generating a 3D human face model, characterized in that, The specific steps of this method for automatically generating 3D face models are as follows: Data loading and processing involves two types of data. The first is loading the user's face image I. usr Secondly, there is the standard face model; User's face image I usr This refers to an image that primarily focuses on the front of a person's face. After loading the image, user facial semantic key points (P) are analyzed. key The detection; A standard face model refers to a 3D model that includes a facial region or other human body parts besides the facial region. The facial region of a standard face model is defined during the design process, and the 3D facial information is M based on the specified facial region. std and texture information I std The texture information of the frontal region of the face is extracted and then projected onto the image to obtain the standard face model projection image I. std_prj Then, project the standard face model onto image I. std_prj Chinese standard face model projection image semantic key points P key_std ; User Model M Based on Computer Vision usr Generate semantic key points P from a standard face model projection map of a single user image and combine it with the face image. key_std Automatically generate user model M usr The user model M usr Contains user texture information I std And the user's facial shape information; Using standard face model projection map semantic key points P key_std Optimizing user model M with facial edge information usr Each user's facial semantic key point P in the face image key All can be in user model M usr Each 3D vertex corresponds to a fixed index in the user model; these 3D vertices are called the 3D keypoints V of the user model. key ; The user model mesh retopology process is to retopologically remodel the user model M. usr The topological structure is reconstructed to form the topological structure of the standard face model. This process utilizes texture information I. std Using the model's 3D information, calculate the 2D and 3D mappings respectively to obtain the retopologically reconstructed user face model; Boundary constraints are added during the mapping calculation process. The mesh vertices are connected to the mapping points, and the angle between the connecting line and the normal plane of the mapping point, the angle between the normal vectors of the two points, and the length of the connecting line are calculated. Thresholds are set for these three values. If they are outside the range, they are considered to be outside the boundary and no mapping is performed. The local shape optimization process of the model mesh is to optimize the deformed mesh of the user face model after retopology based on the surface geometric information of the standard face model mesh. Deformation refers to the sharpness, splitting and overlapping of the mesh. For the vertex of the standard face mesh after retopology, the adjacent points of the point in the original standard face model mesh are rigidly registered to the space after retopology in order to find the correction point of the current point. After mesh optimization is complete, the UV coordinates of the retopologically optimized user face model on the texture image need to be updated. For each optimized mesh vertex, the user model M needs to be found. usr Multiple nearest points are identified, and new UV coordinates are obtained by interpolating the UV coordinates of these nearest points and then assigned to the retopologically reconstructed user face model. The model mesh fusion and stitching process involves fusing the retopologically reconstructed user face model with the standard face model, integrating the retopologically reconstructed model containing user face information into the standard face model to generate the user standard model. Output the result model, which will output the user's standard model.
2. The method for automatically generating a three-dimensional face model according to claim 1, characterized in that: The standard face model contains skeletal and skin information, enabling it to simulate human movements and expressions.
3. The method for automatically generating a three-dimensional face model according to claim 1, characterized in that: The user's facial shape information includes a facial average model T, shape parameter alpha, shape basis S, expression parameter beta, and expression basis E, and a user model M. usr The expression for three-dimensional information is: .
4. The method for automatically generating a three-dimensional face model according to claim 1, characterized in that: The 3D mapping utilizes rigid registration and mesh deformation to increase the overlap between the surfaces of the user's face model and the standard face model in space, thereby finding a mapping relationship to achieve the purpose of retopology. The direction of registration and deformation is from the standard face model to the user's face model. Rigid registration can ensure the maximum overlap between the two models in space without changing the shape of the model.
5. The method for automatically generating a three-dimensional face model according to claim 1, characterized in that: Using the standard face model projection graph semantic key point P of the face image key_std and user model 3D key points V key The projection point V key_proj The error constructs a problem-solving model to solve for the optimal pose R, shape parameter alpha, and expression parameter beta. ； In the above formula, n represents the number of key points involved in the optimization. When the user model M usr If the initial pose shows a profile view that reaches a certain threshold, then the occluded key points will not participate in the optimization. Therefore, the value of n is based on the user model M. usr Initial attitude determined.
6. The method for automatically generating a three-dimensional face model according to claim 1, characterized in that: The 2D mapping utilizes the user model M usr Projected coordinates, user facial semantic key points P key Semantic key points P of standard face model projection map key_std By performing an affine transformation on the projected coordinates in two-dimensional space, the model from the standard model to the user model M can be found. usr The initial mapping.
7. The method for automatically generating a three-dimensional face model according to claim 1, characterized in that: Mesh fusion is achieved by using mesh deformation and mesh smoothing. Mesh smoothing is a smoothing operation for sharp protrusions or depressions in the joints. Mesh deformation is a deformation operation on the unretopologized parts of the model by using the edge points of the retopologized parts as control points.