Face modeling method, device, electronic device and storage medium

Abstract

The present invention provides a face modeling method, device, electronic device and storage medium, wherein the method comprises: obtaining a face point cloud and an initial face mesh, wherein the initial face mesh is obtained based on global principal component analysis modeling; based on the initial face mesh, globally fitting the face point cloud to obtain a first face mesh; based on the first face mesh, performing local principal component analysis modeling on each face region to obtain an initial regional mesh of each face region; based on the initial regional mesh of each face region, performing local fitting on the face point cloud for each face region to obtain a second face mesh. The method, device, electronic device and storage medium provided by the present invention combine global and local principal component analysis to perform three-dimensional face modeling, so as to reduce the error between the second face mesh and the face point cloud, and reduce the mesh deformation when the face point cloud is wrapped based on the second face mesh, thereby greatly reducing the repair time required by the modeler and improving the efficiency of face modeling.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional modeling technology, and in particular to a face modeling method, device, electronic equipment and storage medium. Background Art

[0002] With the development of artificial intelligence technology, three-dimensional (3D) face models have been applied in many scenarios.

[0003] Traditional 3D face modeling includes 3D point reconstruction, triangulation, texture mapping and other steps. When constructing the triangulation, the face point cloud needs to be wrapped with a triangular mesh. Since the mesh shape changes greatly before and after wrapping, the wiring is very different and it is difficult to meet the wiring specifications. Modelers need to spend a lot of time to repair it, which directly affects the efficiency of face modeling. Summary of the invention

[0004] The present invention provides a face modeling method, device, electronic device and storage medium, which are used to solve the defect that three-dimensional face modeling in the prior art is time-consuming.

[0005] The present invention provides a face modeling method, comprising:

[0006] Obtaining a face point cloud and an initial face mesh, wherein the initial face mesh is obtained based on a global principal component analysis model;

[0007] Based on the initial face mesh, globally fit the face point cloud to obtain a first face mesh;

[0008] Based on the first face grid, perform local principal component analysis modeling on each face region to obtain an initial region grid of each face region;

[0009] Based on the initial regional meshes of each face region, local fitting of each face region is performed on the face point cloud to obtain a second face mesh.

[0010] According to a face modeling method provided by the present invention, the method of performing local fitting of each face region on the face point cloud based on the initial region mesh of each face region to obtain a second face mesh comprises:

[0011] Based on the grid points of the initial regional grids of the facial regions and the nearest neighboring points of the grid points of the initial regional grids in the facial point cloud, the constraint value of the local fitting is determined, and the parameters of the local principal component analysis model corresponding to each initial regional grid are adjusted based on the constraint value of the local fitting until the fitting convergence condition is met, thereby obtaining the second facial grid.

[0012] According to a face modeling method provided by the present invention, the determining of the constraint value of the local fitting based on the grid points of the initial regional grid of each face region and the nearest neighboring points of the grid points of the initial regional grid in the face point cloud includes:

[0013] Determining a fitting distance constraint value based on the distance between the grid point and the nearest neighbor point;

[0014] Determining a fitting smoothness constraint value based on a difference in smoothness of the grid points before and after parameter adjustment of the local principal component analysis model;

[0015] A constraint value of the local fitting is determined based on the fitting distance constraint value and the fitting smoothness constraint value.

[0016] According to a face modeling method provided by the present invention, determining a fitting distance constraint value based on the distance between the grid point and the nearest neighbor point includes:

[0017] Determining a first distance constraint value based on the distance between the vertex in the grid point and the nearest neighbor point;

[0018] Determining a second distance constraint value based on the distance between the facial feature points in the grid points and the facial feature points in the facial point cloud;

[0019] The fitting distance constraint value is determined based on the first distance constraint value and the second distance constraint value.

[0020] According to a face modeling method provided by the present invention, determining a fitting smoothness constraint value based on the difference in smoothness of the grid points before and after the parameter adjustment of the local principal component analysis model includes:

[0021] determining a first smoothness of the grid point based on a position of the grid point before a parameter of the local principal component analysis model is adjusted;

[0022] determining a second smoothness of the grid point based on the position of the grid point after the parameters of the local principal component analysis model are adjusted;

[0023] The fitting smoothness constraint value is determined based on a difference between a second smoothness of the grid points and the first smoothness.

[0024] According to a face modeling method provided by the present invention, determining the constraint value of the local fitting based on the fitting distance constraint value and the fitting smoothness constraint value includes:

[0025] Determining a junction constraint value based on the consistency of positions of junction grid points in the initial area grids of each face area, wherein the junction grid points are grid points at the junctions of the initial area grids;

[0026] Determining parameter constraint values ​​based on differences between parameters of the local principal component analysis models corresponding to the initial regional grids;

[0027] Determining the axis constraint value of the region based on the positional relationship between the regional central grid point and the coordinate axis, wherein the regional central grid point is a grid point on the central axis of the face in the initial regional grid;

[0028] The constraint value of the local fitting is determined based on at least one of the boundary constraint value, the parameter constraint value, the axis constraint value in the area, the distance constraint value and the fitting smoothness constraint value.

[0029] According to a face modeling method provided by the present invention, before performing local fitting of each face region on the face point cloud based on the initial region grid of each face region to obtain a second face grid, the method further includes:

[0030] Based on the neck posture model of the face point cloud, posture correction is performed on the initial region mesh of each face region to obtain the posture-corrected initial region mesh, and the posture-corrected initial region mesh is used for local fitting.

[0031] According to a face modeling method provided by the present invention, the step of determining the neck posture model comprises:

[0032] Based on the neck points in the first face mesh, the neck point cloud in the face point cloud is fitted to obtain a neck posture model required for fitting.

[0033] According to a face modeling method provided by the present invention, the method of performing local fitting of each face region on the face point cloud based on the initial region mesh of each face region to obtain a second face mesh comprises:

[0034] Based on the initial regional meshes of each face region, locally fit each face region to the face point cloud aligned with the first face mesh to obtain a second face mesh;

[0035] The facial point cloud aligned with the first facial mesh pose is obtained based on the global fitting.

[0036] According to a face modeling method provided by the present invention, based on the initial region mesh of each face region, the face point cloud is locally fitted to each face region to obtain a second face mesh, and then further comprises:

[0037] The second face mesh and the face point cloud are aligned to obtain a third face mesh.

[0038] According to a face modeling method provided by the present invention, the registering the second face mesh and the face point cloud to obtain a third face mesh includes:

[0039] Based on the grid points of the second face grid and the nearest neighbor points of the grid points of the second face grid in the face point cloud, a registration constraint value is determined, and the positions of the grid points in the second face grid are adjusted based on the registration constraint value until the registration convergence condition is met, so as to obtain the third face grid.

[0040] According to a face modeling method provided by the present invention, determining the registration constraint value based on the grid points of the second face grid and the nearest neighbor points of the grid points of the second face grid in the face point cloud includes:

[0041] Determining a registration distance constraint value based on the distance between the grid point and the nearest neighbor point;

[0042] Determining a registration smoothness constraint value based on a difference in smoothness of the grid points before and after position adjustment;

[0043] The registration constraint value is determined based on the registration distance constraint value and the registration smoothness constraint value.

[0044] According to a face modeling method provided by the present invention, determining the registration constraint value based on the registration distance constraint value and the registration smoothness constraint value includes:

[0045] determining a rigid deformation constraint value based on a transformation relationship of each face in the second face mesh before and after the positions of the mesh points in the second face mesh are adjusted;

[0046] Determining a mouth closing constraint value based on a difference between an edge formed by upper and lower lip vertices in the grid points before and after position adjustment;

[0047] Determining a second central axis constraint value based on a positional relationship between a second central axis grid point and the coordinate axis, wherein the second central axis grid point is a grid point on the central axis of the face in the second face grid;

[0048] The registration constraint value is determined based on at least one of the rigid deformation constraint value, the mouth closure constraint value, the second central axis constraint value, the registration distance constraint value, and the registration smoothness constraint value.

[0049] According to a face modeling method provided by the present invention, the registering the second face mesh and the face point cloud to obtain a third face mesh includes:

[0050] Registering the second face mesh and the face point cloud aligned with the first face mesh to obtain a third face mesh;

[0051] The facial point cloud aligned with the first facial mesh pose is obtained based on the global fitting.

[0052] The present invention also provides a face modeling device, comprising:

[0053] An initial unit, used to obtain a face point cloud and an initial face mesh, wherein the initial face mesh is obtained based on a global principal component analysis model;

[0054] A global fitting unit, configured to perform global fitting on the face point cloud based on the initial face mesh to obtain a first face mesh;

[0055] A local modeling unit, configured to perform local principal component analysis modeling on each face region based on the first face grid to obtain an initial region grid of each face region;

[0056] The local fitting unit is used to perform local fitting of each face region on the face point cloud based on the initial region mesh of each face region to obtain a second face mesh.

[0057] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned face modeling methods is implemented.

[0058] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the face modeling method described in any one of the above is implemented.

[0059] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned face modeling methods.

[0060] The face modeling method, device, electronic device and storage medium provided by the present invention combine global principal component analysis and local principal component analysis to perform three-dimensional face modeling, so as to reduce the error between the second face mesh and the face point cloud, and reduce the mesh deformation when the face point cloud is wrapped based on the second face mesh, thereby greatly reducing the repair time required by the modeler and effectively improving the efficiency of face modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 This is one of the flow charts of the face modeling method provided by the present invention;

[0063] Figure 2 This is the second flow chart of the face modeling method provided by the present invention;

[0064] Figure 3 It is a schematic diagram of the global fitting process provided by the present invention;

[0065] Figure 4 It is a structural schematic diagram of the face modeling device provided by the present invention;

[0066] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] 3D face modeling is a difficult problem in the field of 3D modeling. Modelers are often required to build the face contour and the layout of the facial features, which takes a lot of time. Especially in the mesh construction stage, such as the triangulation stage, the mesh shape before and after wrapping changes greatly, resulting in large wiring differences, making it difficult to meet wiring specifications. Modelers need to spend a lot of time to repair, which directly affects the efficiency of face modeling.

[0069] In view of the above problems, the present invention provides a face modeling method. Figure 1 FIG. 1 is one of the flow charts of the face modeling method provided by the present invention, such as Figure 1 As shown, the method includes:

[0070] Step 110, obtaining a face point cloud and an initial face mesh, wherein the initial face mesh is modeled based on a global principal component analysis.

[0071] Here, the face point cloud refers to the point cloud data obtained by scanning the face of the person to be modeled. The face point cloud is a kind of geometric data representing the shape of the face. It is composed of a series of three-dimensional coordinate points, which describe the shape, size, position and other characteristics of the face. The face point cloud can be obtained in a variety of ways, such as through 3D scanners, depth cameras and other devices. In addition, the face point cloud here can be obtained by scanning the face of the person to be modeled under a neutral expression. The neutral expression refers to the expression of the person in a natural state of closing his mouth and looking straight ahead without any expression.

[0072] The initial face mesh is the initial face mesh template, and the initial face mesh can be obtained by performing a global principal component analysis (Global PCA) on a plurality of face meshes of different lengths and genders. The face meshes of different lengths and genders required here can be obtained through a preset face mesh library. Here, the face meshes stored in the face mesh library can all be face meshes under neutral expressions, and the initial face mesh obtained therefrom is also a face mesh under neutral expressions, that is, the initial face mesh is a neutral face mesh.

[0073] Here, global principal component analysis is a data analysis method for extracting the main features of data and reducing the dimension of data. The global principal component analysis modeling based on various face meshes is a global principal component analysis model for faces. The global principal component analysis model here is a mathematical model, and the initial face mesh can be generated through the global principal component analysis model.

[0074] The initial face mesh obtained in this way can be adapted to various types of faces, thereby reducing the amount of calculation in the subsequent fitting with the face point cloud and improving modeling efficiency.

[0075] Step 120: Based on the initial face mesh, globally fit the face point cloud to obtain a first face mesh.

[0076] Specifically, after obtaining the initial face mesh, the face point cloud can be globally fitted based on the initial face mesh. The global fitting referred to here is actually to fit the face point cloud using the global principal component analysis model corresponding to the initial face mesh. The initial face mesh obtained after global fitting is recorded as the first face mesh. It can be understood that the first face mesh is more closely fitted to the face point cloud than the initial face mesh.

[0077] In the process of fitting the face point cloud, the face point cloud can first be spatially transformed so that the face point cloud can be aligned with the initial face grid; then, the nearest neighbor points of the grid points on the initial face grid on the face point cloud are determined, and based on the distance constraints between the grid points and the nearest neighbor points, the global principal component analysis model corresponding to the initial face grid is solved, and the initial face grid is updated based on the global principal component analysis model, and then a new round of model solving and grid updating is performed until the convergence conditions are met, and the initial face grid that meets the convergence conditions is used as the first face grid.

[0078] Furthermore, based on the initial face mesh, the face point cloud is globally fitted, which can be achieved through the following steps:

[0079] Step i. Calculate the position and posture of the face point cloud in space, and then perform spatial position transformation on the face point cloud based on the position and posture of the face point cloud in space, so that the face point cloud after spatial position transformation can be aligned with the initial face grid, that is, the face point cloud after spatial position transformation and the initial face grid are in the same spatial position, and the center position of the face point cloud after spatial position transformation is the same as the center position of the initial face grid.

[0080] Step ii. Find the nearest neighbor of the grid points on the initial face grid in the face point cloud. Here, the grid points on the initial face grid may specifically include vertices on the initial face grid, and may also include 3D face feature points (3D landmarks) on the initial face grid. The 3D face feature points here may be pre-marked on the initial face grid.

[0081] Step iii. After finding the nearest neighbor point of the grid point on the initial face mesh in the face point cloud, the distance between the grid point and the nearest neighbor point can be used as a constraint to solve the parameters of the global principal component analysis model corresponding to the initial face mesh, and based on the parameters of the solved global principal component analysis model, the initial face mesh is updated.

[0082] Step iv. Based on the updated initial face mesh, you can return to re-execute steps i-iii until the constraints in step iii converge. The initial face mesh updated based on the parameters of the converged global principal component analysis model is recorded as the first face mesh.

[0083] Step 130: Based on the first face mesh, perform local principal component analysis modeling on each face region to obtain an initial region mesh of each face region.

[0084] Specifically, after the global fitting is completed, local principal component analysis (Local PCA) modeling can be performed on each face region based on the first face mesh obtained by the global fitting.

[0085] Here, local principal component analysis is a data analysis method that extracts the main features of data and reduces the data dimension. It captures local features and trends in the data by performing principal component analysis in the local range of the data. Unlike global principal component analysis (Global PCA), local principal component analysis pays more attention to the changes and structures of data in the local range.

[0086] Specifically, in the embodiment of the present invention, considering the differences in the structural shapes of various regions of the face, the face is divided into multiple face regions, and a local principal component analysis is performed on each face region, thereby obtaining a local principal component analysis model for each face region. The face regions here may be divided into forehead, eyes, nose, lips, cheeks, chin, etc., which is not specifically limited in the embodiment of the present invention.

[0087] It can be understood that each face region corresponds to a local principal component analysis model, and an initial region grid of each face region can be generated based on the local principal component analysis model corresponding to each face region.

[0088] Furthermore, local principal component analysis modeling is performed on each facial region based on the first facial grid. Specifically, sparse sampling can be performed on the first facial grid, and based on the grids of each facial region obtained by sparse sampling, the local principal component analysis model corresponding to each facial region is solved to obtain the initial parameter values ​​of the local principal component analysis model of each facial region, thereby generating an initial region grid corresponding to each facial region.

[0089] It can be understood that the initial regional mesh of each face area is constructed based on the first face mesh obtained by global fitting of the face point cloud. Therefore, the initial regional mesh of each face area itself is more closely fitted to the face point cloud, which can reduce the amount of calculation in the subsequent fitting with the face point cloud and improve the modeling efficiency.

[0090] Step 140: Based on the initial regional meshes of the respective face regions, local fitting of the face point cloud is performed on the respective face regions to obtain a second face mesh.

[0091] Specifically, after obtaining the initial regional mesh of each face region, local fitting of each face region can be performed on the face point cloud based on the initial regional mesh. The local fitting referred to here is actually to apply the local principal component analysis model corresponding to the initial regional mesh of each face region to fit the point cloud of each face region in the face point cloud. Thus, the initial regional mesh of each face region after local fitting is obtained to form a complete face mesh, which is recorded as the second face mesh here. It can be understood that the mesh of each face region in the second face mesh is more closely fitted to the face point cloud than the first face mesh.

[0092] In the process of fitting the facial point cloud, the nearest neighbor points of the grid points on the initial regional grid of each facial region on the facial point cloud can be determined, and based on the distance constraint between the grid point and the nearest neighbor point, the local principal component analysis model corresponding to the initial regional grid is solved, and the initial regional grid is updated based on the local principal component analysis model, and then a new round of model solving and grid updating is performed again until the convergence condition is met, and the initial regional grids of each facial region that meet the convergence condition are combined as the second facial grid.

[0093] Here, in the process of solving the model and updating the grid, in addition to considering the distance constraints between the grid points and the nearest neighboring points, the position constraints of the edge grid points of the intersecting face areas can also be considered to ensure that the initial area grids of the intersecting face areas can be connected to form a smooth second face grid. The embodiment of the present invention does not make specific limitations on this.

[0094] The second face mesh thus obtained can be used as a wiring template for three-dimensional face modeling, thereby realizing three-dimensional modeling of the face point cloud.

[0095] The method provided by the embodiment of the present invention combines global principal component analysis and local principal component analysis to perform three-dimensional face modeling, so as to reduce the error between the second face mesh and the face point cloud, and reduce the mesh deformation when the face point cloud is wrapped based on the second face mesh, thereby greatly reducing the repair time required by the modeler and effectively improving the efficiency of face modeling.

[0096] Moreover, according to the application of the method provided in the embodiment of the present invention, 3D face modeling based on the method provided in the embodiment of the present invention can compress the neutral face modeling that conventionally requires a modeler to spend 4 or 5 days to complete within 2 or 3 days, thereby greatly improving the efficiency of face modeling.

[0097] Based on the above embodiment, in step 140, based on the initial regional mesh of each face region, the face point cloud is locally fitted to each face region to obtain a second face mesh, including:

[0098] Based on the grid points of the initial regional grids of the facial regions and the nearest neighboring points of the grid points of the initial regional grids in the facial point cloud, the constraint value of the local fitting is determined, and the parameters of the local principal component analysis model corresponding to each initial regional grid are adjusted based on the constraint value of the local fitting until the fitting convergence condition is met, thereby obtaining the second facial grid.

[0099] Specifically, in the process of local fitting, the nearest neighbor points of the grid points on the initial grid of each face region on the face point cloud can be first determined, and the constraint value of the local fitting is calculated based on the grid points and the nearest neighbor points, and the constraint value is applied to solve the parameters of the local principal component analysis model corresponding to each face region to achieve parameter iteration of the local principal component analysis model, and the initial region grid of each face region is updated based on the local principal component analysis model of each face region after parameter iteration, and then a new round of model parameter iteration and grid update is performed again until the fitting convergence condition is met, and the initial region grids that meet the fitting convergence condition are combined into a second face grid. The fitting convergence condition here can be that the constraint value of the local fitting is less than or equal to a preset threshold, or that the number of parameter iterations in the local fitting reaches the target number, and the embodiment of the present invention does not specifically limit this.

[0100] Furthermore, based on the initial regional grid of each face region, the face point cloud is locally fitted, which can be specifically achieved through the following steps:

[0101] Step I. Find the nearest neighbor points of the grid points on the initial regional grid of each face region in the face point cloud. Here, the grid points on the initial regional grid may specifically include vertices on the initial regional grid, and may also include 3D face feature points on the initial regional grid.

[0102] Step II. After finding the nearest neighboring points of the grid points on the initial regional grid in the face point cloud, the local fitting constraint values ​​of the grid points and the nearest neighboring points can be calculated to solve the parameters of the local principal component analysis model corresponding to the initial regional grid, and the initial regional grid can be updated based on the parameters of the local principal component analysis model obtained by solving the local principal component analysis model.

[0103] Step III. Based on the updated initial region mesh, you can return to re-execute steps I-II until the fitting convergence condition is met, and the initial region mesh updated based on the parameters of the converged local principal component analysis model forms a second face mesh.

[0104] Among them, the constraint value of local fitting can be based on the distance constraint between the grid point and the nearest neighbor point, and can also consider the position constraint of the edge grid point of the intersecting face area to ensure that the initial area grid of the intersecting face area can be connected to form a smooth second face grid. The embodiment of the present invention does not make specific limitations on this.

[0105] Based on any of the above embodiments, in step 140, determining the constraint value of the local fitting based on the grid points of the initial regional grid of each face region and the nearest neighbor points of the grid points of the initial regional grid in the face point cloud includes:

[0106] Determining a fitting distance constraint value based on the distance between the grid point and the nearest neighbor point;

[0107] Determining a fitting smoothness constraint value based on a difference in smoothness of the grid points before and after parameter adjustment of the local principal component analysis model;

[0108] A constraint value of the local fitting is determined based on the fitting distance constraint value and the fitting smoothness constraint value.

[0109] Specifically, for the calculation of the constraint value of the local fitting, the fitting distance constraint value and the fitting smoothness constraint value may be calculated.

[0110] Among them, the fitting distance constraint value represents the constraint term formed by the distance between the grid points in the initial area grid and the nearest neighbor points in the face point cloud in the local fitting. Further, the fitting distance constraint value can reflect the distance between the grid points and the nearest neighbor points. It can be understood that the farther the distance between each grid point and the corresponding nearest neighbor point, the larger the fitting distance constraint value, and the closer the distance between each grid point and the corresponding nearest neighbor point, the smaller the fitting distance constraint value.

[0111] It can be understood that using the fitting distance constraint value as a part of the local fitting constraint value can ensure that during the local fitting process, the initial regional mesh is iteratively updated in the direction of fitting the face point cloud.

[0112] The fitting smoothness constraint value represents the constraint term formed by the difference in smoothness of the grid points in the initial regional grid caused by the parameter adjustment of the local principal component analysis model in the local fitting. It can be understood that the greater the difference in smoothness of each grid point before and after the parameter adjustment of the local principal component analysis model, the greater the fitting smoothness constraint value; the smaller the difference in smoothness of each grid point before and after the parameter adjustment of the local principal component analysis model, the smaller the fitting smoothness constraint value. Here, the smoothness of the grid point can be obtained by performing a Laplace transform on the coordinates of the grid point.

[0113] By using the fitting smoothness constraint value as a part of the constraint value of the local fitting, it can be ensured that during the local fitting process, the smoothness of the local wiring in the initial regional mesh can be consistent with the smoothness of the local wiring in the initial regional mesh before the update in step 130, so that the second face mesh obtained after the local fitting converges will not have a large deformation, thereby ensuring the wiring standardization based on the second face mesh, avoiding the problem that the modeler needs to spend a lot of time to repair the second face mesh due to non-compliance with the wiring standard, thereby improving the efficiency of 3D face modeling.

[0114] After obtaining the fitting distance constraint value and the fitting smoothness constraint value, the two may be added together as the local fitting constraint value, or the two may be weightedly summed to obtain the local fitting constraint value, which is not specifically limited in the embodiment of the present invention.

[0115] Based on any of the above embodiments, in step 140, determining the fitting distance constraint value based on the distance between the grid point and the nearest neighbor point includes:

[0116] Determining a first distance constraint value based on the distance between the vertex in the grid point and the nearest neighbor point;

[0117] Determining a second distance constraint value based on the distance between the facial feature points in the grid points and the facial feature points in the facial point cloud;

[0118] The fitting distance constraint value is determined based on the first distance constraint value and the second distance constraint value.

[0119] Specifically, the fitting distance constraint value can be divided into two parts, one of which is the distance between the vertices in the initial area mesh and the corresponding nearest neighbor points in the face point cloud, and the other is the distance between the 3D face feature points in the initial area mesh and the 3D face feature points in the face point cloud.

[0120] Among them, the sum of the distances between each vertex in the initial area grid and the corresponding nearest neighbor point in the face point cloud can be used as the first part of the fitting distance constraint value; the sum of the distances between each 3D face feature point in the initial area grid and the 3D face feature point under the same semantics in the face point cloud can be used as the second part of the fitting distance constraint value.

[0121] On this basis, the first part and the second part may be added together as the fitting distance constraint value, or the first part and the second part may be weightedly summed to obtain the fitting distance constraint value, which is not specifically limited in the embodiment of the present invention.

[0122] Based on any of the above embodiments, in step 140, determining the fitting smoothness constraint value based on the difference in smoothness of the grid points before and after the parameter adjustment of the local principal component analysis model includes:

[0123] determining a first smoothness of the grid point based on a position of the grid point before a parameter of the local principal component analysis model is adjusted;

[0124] determining a second smoothness of the grid point based on the position of the grid point after the parameters of the local principal component analysis model are adjusted;

[0125] The fitting smoothness constraint value is determined based on a difference between a second smoothness of the grid points and the first smoothness.

[0126] Specifically, in the calculation process of the fitting smooth constraint value, the smoothness of the grid points in the initial regional grid of each face region determined in step 130 can be used as the first smoothness, and the smoothness of the grid points in the initial regional grid after each local principal component analysis model parameter adjustment can be used as the second smoothness. For each grid point, by calculating the difference between the second smoothness and the first smoothness, the difference in smoothness of the grid point before and after the parameter adjustment of the local principal component analysis model can be obtained. The first smoothness and the second smoothness here can be obtained by performing Laplace transform on the coordinates of the grid point.

[0127] It can be understood that the greater the difference between the second smoothness and the first smoothness of each grid point, the greater the difference in smoothness of the grid point before and after the parameter adjustment of the local principal component analysis model, and the larger the fitting smoothness constraint value; the smaller the difference between the second smoothness and the first smoothness of each grid point, the smaller the difference in smoothness of the grid point before and after the parameter adjustment of the local principal component analysis model, and the smaller the fitting smoothness constraint value.

[0128] Furthermore, the fitting smoothness constraint value may be obtained by accumulating the difference between the second smoothness and the first smoothness of the grid points in each initial regional grid.

[0129] Based on any of the above embodiments, in step 140, determining the constraint value of the local fitting based on the fitting distance constraint value and the fitting smoothness constraint value includes:

[0130] Determining a junction constraint value based on the consistency of positions of junction grid points in the initial area grids of each face area, wherein the junction grid points are grid points at the junctions of the initial area grids;

[0131] Determining parameter constraint values ​​based on differences between parameters of the local principal component analysis models corresponding to the initial regional grids;

[0132] Determining the axis constraint value of the region based on the positional relationship between the regional central grid point and the coordinate axis, wherein the regional central grid point is a grid point on the central axis of the face in the initial regional grid;

[0133] The constraint value of the local fitting is determined based on at least one of the boundary constraint value, the parameter constraint value, the axis constraint value in the area, the distance constraint value and the fitting smoothness constraint value.

[0134] Specifically, for the calculation of the constraint value of local fitting, in addition to calculating the fitting distance constraint value and the fitting smoothness constraint value, at least one of the boundary constraint value, the parameter constraint value, and the regional axis constraint value can also be calculated.

[0135] The boundary constraint value represents the grid points at the boundary between the initial regional grids in the local fitting, that is, the consistency of the positions of the boundary grid points in the initial regional grids. It can be understood that at the boundary of multiple initial regional grids, there are boundary grid points in each initial regional grid, and in order to ensure the integrity of the initial regional grids of different face regions, that is, the smoothness of the initial regional grids of different face regions at the boundary, the positions of the same boundary grid points in the initial regional grids of different face regions should be consistent.

[0136] Therefore, for the intersection grid points, the positions of the intersection grid points in each initial area grid can be weighted summed to obtain the intersection position of the intersection grid points, and the distance between the position of the intersection grid points in each initial area grid and the above-mentioned intersection position is calculated, and the distance between the position of the intersection grid points in each initial area grid and the above-mentioned intersection position is accumulated as the intersection constraint value.

[0137] It can be understood that the greater the position difference of the same boundary grid point in the initial area grids of different face areas, the worse the consistency of the position of the boundary grid point in the initial area grids of each face area, and the larger the boundary constraint value; the smaller the position difference of the same boundary grid point in the initial area grids of different face areas, the better the consistency of the position of the boundary grid point in the initial area grids of each face area, and the smaller the boundary constraint value.

[0138] By using the boundary constraint value as a part of the constraint value of the local fitting, it can be ensured that during the local fitting process, the positions of the same boundary grid points in the initial regional grids of different face regions are consistent, thereby ensuring the integrity of the initial regional grids of each face region.

[0139] The parameter constraint value represents the difference between the parameters of the local principal component analysis model corresponding to each initial regional grid. Specifically, the parameters of the local principal component analysis model corresponding to each initial regional grid can be regularized, and the regularization result is used as the parameter constraint value reflecting the difference between the parameters.

[0140] It can be understood that the greater the difference between the parameters of each local principal component analysis model, the greater the parameter constraint value; and the smaller the difference between the parameters of each local principal component analysis model, the smaller the parameter constraint value.

[0141] By taking the parameter constraint value as a part of the constraint value of the local fitting, the difference between the parameters of each local principal component analysis model can be constrained to avoid the outlier of the parameters of a local principal component analysis model.

[0142] The regional axis constraint value represents the grid points on the central axis of the face determined based on each initial regional grid, that is, whether the regional central axis grid points are located on a preset coordinate axis. For example, assuming that the preset coordinate axis is the y-axis, the positional relationship between the regional central axis grid points and the y-axis can be determined, that is, whether the regional central axis grid points are located on the y-axis.

[0143] It can be understood that the central axis grid points in the region constitute the central axis of the face, and based on the positional relationship between the central axis grid points in the region and the y-axis, it can be determined whether the central axis of the face is on the y-axis. The more the central axis of the face deviates from the y-axis, the greater the constraint value of the central axis in the region; the closer the central axis of the face is to the y-axis, the smaller the constraint value of the central axis in the region.

[0144] By taking the regional central axis constraint value as a part of the local fitting constraint value, it can be ensured that the central axis of the face coincides with the coordinate axis during the local fitting process, that is, the face mesh formed based on each initial regional mesh is a positive mesh.

[0145] After obtaining the fitting distance constraint value and the fitting smoothness constraint value, as well as at least one of the boundary constraint value, the parameter constraint value, and the central axis constraint value, the above-mentioned various constraint values ​​can be added as the local fitting constraint value, or the above-mentioned various constraint values ​​can be weightedly summed to obtain the local fitting constraint value. The embodiment of the present invention does not make specific limitations on this.

[0146] Based on any of the above embodiments, in step 140, based on the initial regional mesh of each face region, the face point cloud is locally fitted for each face region to obtain a second face mesh, and the process also includes:

[0147] Based on the neck posture model of the face point cloud, posture correction is performed on the initial region mesh of each face region to obtain the posture-corrected initial region mesh, and the posture-corrected initial region mesh is used for local fitting.

[0148] Specifically, considering that the neck of a person is not completely in a neutral posture when collecting face point clouds, and the face mesh is constructed based on the neutral posture, in order to improve the fitting reliability, the neck posture model of the face point cloud can be obtained, and based on this, the initial regional mesh of each face area obtained by local principal component analysis modeling is posture corrected. After posture correction, the neck points in the initial regional mesh of each face area can be closer to the actual posture of the neck in the face point cloud, thereby improving the reliability of subsequent local fitting.

[0149] Here, the neck posture model is used to transform the posture of the neck. Specifically, it can be to transform the neck posture of the first face mesh so that the neck points in the first face mesh are closer to the mapping relationship of the actual posture of the neck in the face point cloud. Considering that the initial area mesh of each face area is modeled on the basis of the first face mesh, the neck posture model can be applied to the neck posture correction of the initial area mesh of each face area.

[0150] It can be understood that the initial regional mesh of each face region after the neck posture correction is completed can be directly used as the initial regional mesh of each face region for local fitting in step 140, thereby obtaining a second face mesh that is more consistent with the face point cloud.

[0151] Based on any of the above embodiments, the step of determining the neck posture model includes:

[0152] Based on the neck points in the first face mesh, the neck point cloud in the face point cloud is fitted to obtain a neck posture model required for fitting.

[0153] Specifically, considering that the neck posture in the face point cloud is not completely a neutral posture, the first face mesh and face point cloud with aligned postures can be fitted on the neck point. Here, the weights of the bones and skin of the neck can be set in advance, and in the fitting process, the neck point in the first face mesh and the nearest neighbor point in the neck point cloud in the face point cloud are used as constraints to fit the above-mentioned pre-set weights of the bones and skin. In this fitting process, the neck point in the first face mesh can be rotated, translated, etc., so that the neck point in the face mesh can be continuously close to the neck point cloud in the face point cloud until convergence.

[0154] For ease of explanation, the first face mesh obtained by neck point fitting convergence is recorded as the corrected face mesh. After obtaining the corrected face mesh, the conversion relationship from the first face mesh to the corrected face mesh can be calculated based on the first face mesh and the corrected face mesh, that is, the neck posture model required for neck point fitting.

[0155] Based on any of the above embodiments, in step 140, performing local fitting of each face region on the face point cloud based on the initial region mesh of each face region to obtain a second face mesh includes:

[0156] Based on the initial regional meshes of each face region, locally fit each face region to the face point cloud aligned with the first face mesh to obtain a second face mesh;

[0157] The facial point cloud aligned with the first facial mesh pose is obtained based on the global fitting.

[0158] Specifically, in step 120, in the process of global fitting based on the initial face mesh and the face point cloud, after each iterative update of the initial face mesh, the face point cloud is spatially transformed to achieve alignment of the initial face mesh and the face point cloud. Thus, after completing the global fitting, not only the converged first face mesh can be obtained, but also the face point cloud that is aligned with the first face mesh after spatial transformation can be obtained.

[0159] Therefore, considering that the initial regional grid of each face area is obtained on the basis of the first face grid, when performing local fitting of each face area with the face point cloud based on the initial regional grid of each face area, the face point cloud aligned with the first face grid posture can be directly applied, that is, the face point cloud aligned with the first face grid posture can be locally fitted for each face area based on the initial regional grid of each face area.

[0160] Based on any of the above embodiments, after step 140, the method further includes:

[0161] The second face mesh and the face point cloud are aligned to obtain a third face mesh.

[0162] Specifically, after obtaining the second face mesh, point cloud registration can be performed on the second mesh face and the face point cloud, for example, non-rigid point cloud registration (Non-Rigid Iterative Closest Point, Non-rigidICP). Through point cloud registration, the template obtained through the above steps, i.e., the second face mesh, can be fitted to the face point cloud.

[0163] Here, the second face mesh obtained after the second face mesh and the face point cloud are registered is recorded as the third face mesh.

[0164] Furthermore, the method for aligning the second face mesh and the face point cloud can be to find the nearest neighbor points of the grid points in the second face mesh in the face point cloud, use the grid points and the nearest neighbor points as constraints, solve the new positions of each grid point in the second face mesh, and adjust the positions of each grid point in the second face mesh so that the second face mesh can better fit the face point cloud, thereby obtaining a third face mesh.

[0165] Furthermore, for the neck posture model based on the face point cloud in the above embodiment, after the initial area mesh of each face area is posture-corrected, a second face mesh is obtained by local fitting of each face area. When the second face mesh is aligned with the face point cloud in the embodiment of the present invention, the neck point in the second face mesh has been corrected, which can overcome the problem of non-neutral neck posture. Therefore, the neck point can also be involved in the alignment. That is, the third face mesh obtained by the alignment here can achieve the alignment of the face and the neck.

[0166] Based on any of the foregoing embodiments, the registering the second face mesh and the face point cloud to obtain a third face mesh includes:

[0167] Based on the grid points of the second face grid and the nearest neighbor points of the grid points of the second face grid in the face point cloud, a registration constraint value is determined, and the positions of the grid points in the second face grid are adjusted based on the registration constraint value until the registration convergence condition is met, so as to obtain the third face grid.

[0168] Specifically, for the point cloud registration of the second face mesh, the registration constraint value can be calculated based on the mesh points in the second face mesh and the nearest neighbor points of the mesh points in the second face mesh in the face point cloud.

[0169] The mesh points here may include face mesh points and neck mesh points.

[0170] The alignment constraint value here can be calculated based on the distance constraint between the grid point and the nearest neighbor point, or the difference in smoothness of the grid before and after the grid point position is adjusted can be considered, and whether the deformation of each face on the grid after the grid point position is adjusted is rigid can also be considered. The embodiment of the present invention does not make specific limitations on this.

[0171] After calculating the registration constraint value, the position of each grid point in the second face grid can be adjusted under the constraint of the registration constraint value. After the position adjustment, the nearest neighbor point of the grid point in the face point cloud can be re-determined, and a new registration constraint value can be calculated based on this, and the position of each grid point in the second face grid can be adjusted again until the registration convergence condition is met.

[0172] The registration convergence condition here may be that the registration constraint value is less than or equal to a preset threshold, or that the number of position adjustments in the point cloud registration reaches a target number, which is not specifically limited in the embodiment of the present invention.

[0173] Based on any of the above embodiments, in the process of point cloud registration, determining the registration constraint value based on the grid points of the second face grid and the nearest neighbor points of the grid points of the second face grid in the face point cloud includes:

[0174] Determining a registration distance constraint value based on the distance between the grid point and the nearest neighbor point;

[0175] Determining a registration smoothness constraint value based on a difference in smoothness of the grid points before and after position adjustment;

[0176] The registration constraint value is determined based on the registration distance constraint value and the registration smoothness constraint value.

[0177] Specifically, with respect to the calculation of the registration constraint value, a registration distance constraint value and a registration smoothness constraint value may be calculated.

[0178] Among them, the registration distance constraint value represents the constraint item formed by the distance between the grid point in the second face grid and the nearest neighbor point in the face point cloud in point cloud registration. Further, the registration distance constraint value can reflect the distance between the grid point and the nearest neighbor point. It can be understood that the farther the distance between each grid point and the corresponding nearest neighbor point, the larger the registration distance constraint value, and the closer the distance between each grid point and the corresponding nearest neighbor point, the smaller the registration distance constraint value.

[0179] It can be understood that using the registration distance constraint value as a part of the registration constraint value can ensure that during the point cloud registration process, the second face mesh is iteratively updated in a direction that fits the face point cloud.

[0180] Furthermore, the registration distance constraint value can also be divided into two parts, one of which is the distance between the vertices in the second face mesh and the corresponding nearest neighbor points in the face point cloud, and the other is the distance between the 3D face feature points in the second face mesh and the 3D face feature points in the face point cloud.

[0181] Among them, the sum of the distances between each vertex in the second face mesh and the corresponding nearest neighbor points in the face point cloud can be used as the first part of the registration distance constraint value; the sum of the distances between each 3D face feature point in the second face mesh and the 3D face feature points with the same semantics in the face point cloud can be used as the second part of the registration distance constraint value.

[0182] On this basis, the first part and the second part can be added as the registration distance constraint value, or the first part and the second part can be weighted and summed to obtain the registration distance constraint value. Preferably, considering that the distance between the second face mesh itself and the face point cloud is very close, in the calculation scheme of the registration distance constraint value, only the first part can be calculated, or the weight of the second part can be set very small.

[0183] The registration smoothness constraint value represents the constraint term formed by the difference in the smoothness of the grid points before and after the grid point position adjustment of the second face grid in the point cloud registration. It can be understood that the greater the difference in the smoothness of each grid point before and after the grid point position adjustment, the greater the registration smoothness constraint value; the smaller the difference in the smoothness of each grid point before and after the grid point position adjustment, the smaller the registration smoothness constraint value. Here, the smoothness of the grid point can be obtained by performing a Laplace transform on the coordinates of the grid point.

[0184] By using the registration smoothness constraint value as a part of the constraint value of the point cloud registration, it can be ensured that during the point cloud registration process, the smoothness of the wiring in the second face mesh can be consistent with the smoothness of the wiring in the second face mesh obtained in step 140, so that the third face mesh obtained after the point cloud registration converges will not have a large deformation, thereby ensuring the standardization of the wiring based on the third face mesh, avoiding the problem that the modeler needs to spend a lot of time to repair the third face mesh due to non-compliance with the wiring specifications, thereby improving the efficiency of 3D face modeling.

[0185] Furthermore, in the process of calculating the registration smoothness constraint value, the smoothness of the grid points in the second face grid obtained in step 140 can be used as the first smoothness, and the smoothness of the grid points in the second face grid after each grid point position adjustment can be used as the second smoothness. For each grid point, by calculating the difference between the second smoothness and the first smoothness, the difference in smoothness of the grid point before and after the grid point position adjustment can be obtained. The first smoothness and the second smoothness here can be obtained by performing Laplace transform on the coordinates of the grid point.

[0186] It can be understood that the greater the difference between the second smoothness and the first smoothness of each grid point, the greater the difference in smoothness of the grid point before and after the grid point position is adjusted, and the greater the registration smoothness constraint value; the smaller the difference between the second smoothness and the first smoothness of each grid point, the smaller the difference in smoothness of the grid point before and after the grid point position is adjusted, and the smaller the registration smoothness constraint value.

[0187] For example, the registration smoothness constraint value may be obtained by accumulating the difference between the second smoothness and the first smoothness of the grid points in the second face grid.

[0188] After obtaining the registration distance constraint value and the registration smoothness constraint value, the two can be added as the registration constraint value of the point cloud registration, or the two can be weightedly summed to obtain the registration constraint value of the point cloud registration, which is not specifically limited in the embodiment of the present invention.

[0189] Based on any of the above embodiments, in the process of point cloud registration, determining the registration constraint value based on the registration distance constraint value and the registration smoothness constraint value includes:

[0190] determining a rigid deformation constraint value based on a transformation relationship of each face in the second face mesh before and after the positions of the mesh points in the second face mesh are adjusted;

[0191] Determining a mouth closing constraint value based on a difference between an edge formed by upper and lower lip vertices in the grid points before and after position adjustment;

[0192] Determining a second central axis constraint value based on a positional relationship between a second central axis grid point and the coordinate axis, wherein the second central axis grid point is a grid point on the central axis of the face in the second face grid;

[0193] The registration constraint value is determined based on at least one of the rigid deformation constraint value, the mouth closure constraint value, the second central axis constraint value, the registration distance constraint value, and the registration smoothness constraint value.

[0194] Specifically, for the calculation of the registration constraint value, in addition to calculating the registration distance constraint value and the registration smoothness constraint value, at least one of the rigid deformation constraint value, the mouth closure constraint value, and the second central axis constraint value may also be calculated.

[0195] The rigid deformation constraint value represents the magnitude of the transformation relationship of each face on the second face mesh before and after the grid point position is adjusted in the point cloud registration. It is understandable that, in the point cloud registration process, the adjustment of the grid point position on the second face mesh will drive the deformation on the second face mesh, and thus, for any face, the change relationship of the face is the transformation relationship required to convert the face from the face in the second face mesh determined in step 140 to the face after the grid point position is adjusted.

[0196] During the point cloud registration process, it is necessary to ensure that the deformation of each face on the second face mesh is a rigid deformation, especially the deformation of the faces in the eye sockets, the mouth, the back of the head and other areas is a rigid deformation, so as to avoid the third face mesh from being greatly deformed after the point cloud registration, and ensure the authenticity of the third face mesh and avoid distortion. To this end, it is necessary to constrain the difference between the transformation relationship of each face in the second face mesh before and after the grid point position is adjusted and the unit matrix. For example, the difference between the change relationship of each face and the unit matrix can be accumulated as the rigid deformation constraint value.

[0197] It can be understood that the greater the difference between the transformation relationship of each surface and the unit matrix, the greater the deformation of each surface before and after the grid point position is adjusted, and the larger the rigid deformation constraint value; the smaller the difference between the transformation relationship of each surface and the unit matrix, the smaller the deformation of each surface before and after the grid point position is adjusted, and the smaller the rigid deformation constraint value.

[0198] By using the rigid deformation constraint value as part of the constraint value of point cloud registration, it can be ensured that during the point cloud registration process, the deformation of each surface of the second face mesh remains rigid as much as possible, so that the third face mesh obtained by point cloud registration can retain the details and structural information of complex areas such as the inside of the eye socket and the inside of the mouth, thereby ensuring the realism of the third face mesh.

[0199] The closed mouth constraint value represents the difference in the edge formed by the upper and lower lip vertices in the second face mesh before and after the grid point position is adjusted in the point cloud registration. It can be understood that the second face mesh determined based on step 140 is a neutral face mesh of the person in the closed mouth state. In order to ensure that in the point cloud registration process, even if it is affected by the adjustment of the grid point position, the third face mesh in the closed mouth state can still be obtained, the change of the edge formed by the upper and lower lip vertices in the second face mesh before and after the grid point position is adjusted can be used as the closed mouth constraint.

[0200] For example, pairs of upper and lower lip vertices can be selected from the inner lip points to construct an edge. The length of the edge in the second face mesh determined in step 140 is subtracted from the length of the edge after the position point is adjusted during the point cloud alignment process. The difference in the length of the edge constructed by each pair of upper and lower lip vertices before and after the position point is adjusted is accumulated as the closing mouth constraint value.

[0201] By using the closed mouth constraint value as a part of the constraint value of the point cloud registration, it can be ensured that the third face mesh obtained based on the point cloud registration is a neutral face mesh of the person in the closed mouth state, so as to facilitate face registration.

[0202] The second central axis constraint value represents the grid point on the central axis of the face determined based on the second face grid, that is, whether the second central axis grid point is located on a preset coordinate axis. For example, assuming that the preset coordinate axis is the y-axis, the positional relationship between the second central axis grid point and the y-axis can be determined, that is, whether the second central axis grid point is located on the y-axis.

[0203] It can be understood that the second central axis grid points constitute the central axis of the face, and based on the positional relationship between the second central axis grid points and the y-axis, it can be determined whether the central axis of the face is on the y-axis. The more the central axis of the face deviates from the y-axis, the greater the constraint value of the second central axis; the closer the central axis of the face is to the y-axis, the smaller the constraint value of the second central axis.

[0204] By using the second central axis constraint value as a part of the constraint value of point cloud registration, it can be ensured that during the point cloud registration process, the central axis of the face coincides with the coordinate axis, that is, the third face mesh obtained thereby is a positive mesh.

[0205] After obtaining the registration distance constraint value and the registration smoothness constraint value, as well as at least one of the rigid deformation constraint value, the closed mouth constraint value, and the second central axis constraint value, the above-mentioned types of constraint values ​​can be added to the registration constraint value, or the above-mentioned types of constraint values ​​can be weightedly summed to obtain the registration constraint value. The embodiment of the present invention does not make any specific limitations on this.

[0206] Based on any of the above embodiments, after step 140, registering the second face mesh and the face point cloud to obtain a third face mesh includes:

[0207] Registering the second face mesh and the face point cloud aligned with the first face mesh to obtain a third face mesh;

[0208] The facial point cloud aligned with the first facial mesh pose is obtained based on the global fitting.

[0209] Specifically, in step 120, in the process of global fitting based on the initial face mesh and the face point cloud, after each iterative update of the initial face mesh, the face point cloud is spatially transformed to achieve alignment of the initial face mesh and the face point cloud. Thus, after completing the global fitting, not only the converged first face mesh can be obtained, but also the face point cloud that is aligned with the first face mesh after spatial transformation can be obtained.

[0210] Therefore, considering that the second face mesh is obtained based on the first face mesh, when performing point cloud registration on the second face mesh and the face point cloud, the face point cloud aligned with the first face mesh posture can be directly applied, that is, the second face mesh and the face point cloud aligned with the first face mesh posture can be point cloud fitted.

[0211] Based on any of the above embodiments, Figure 2 FIG. 2 is a flow chart of the face modeling method provided by the present invention. Figure 2 As shown, the face modeling method may include the following steps:

[0212] Step 210, obtaining a face point cloud with 3D face feature points:

[0213] The face point cloud can be scanned and obtained. According to the definition of 3D face feature points of common templates, the corresponding 3D face feature points are marked on the face point cloud. Here, the marking of 3D face feature points can be performed by a modeler or realized by an algorithm.

[0214] Step 220, global + local fitting point cloud:

[0215] After obtaining the face point cloud, the initial face mesh can be obtained through global principal component analysis modeling, and the face point cloud can be globally fitted based on the initial face mesh to obtain the first face mesh. Subsequently, local principal component analysis modeling is performed on each face region based on the first face mesh to obtain the initial regional mesh of each face region. Then, based on the initial regional mesh of each face region, the face point cloud is locally fitted to each face region to obtain the second face mesh, thereby achieving global + local point cloud fitting.

[0216] In this process, Figure 3 is a schematic diagram of the global fitting process provided by the present invention; Figure 3 As shown in Figure 2, the global fitting process includes the following steps:

[0217] Step 310, based on the position and posture of the face point cloud, update the point cloud position:

[0218] The position and posture of the face point cloud in space are calculated, and then the face point cloud is transformed in space based on the position and posture of the face point cloud in space, so that the face point cloud can be aligned with the initial face grid.

[0219] Step 320, update and solve the global principal component analysis model parameters:

[0220] The parameters of the global principal component analysis model corresponding to the initial face mesh can be solved, and the initial face mesh under the current parameters obtained by the solution can be updated. In the solution process, the nearest neighbor point constraint and the 3D face feature point constraint can be referred to.

[0221] Step 330, find the nearest neighbor point of the face mesh in the face point cloud under the current parameters:

[0222] For the face mesh under the current parameters obtained in the previous step, that is, the initial face mesh after iterative update, the nearest neighbor points of the mesh points in the face point cloud can be found to serve as constraints when updating and solving the global principal component analysis model parameters.

[0223] After the search is completed, the process may return to step 310 until the above steps 310 - 330 converge after multiple rounds of iterations.

[0224] Step 340, iterate several times, and output the first face mesh and the aligned face point cloud:

[0225] Here, after several iterations, if the convergence condition is met, the initial face mesh after iteration can be output, which is recorded as the first face mesh. Since step 310 is executed each time in the iteration, a face point cloud aligned with the first face mesh can be obtained.

[0226] Step 230, point cloud registration:

[0227] After completing the global + local fitting point cloud and obtaining the second face mesh, the second face mesh can be applied to the face point cloud for point cloud registration, and the second face mesh obtained by the point cloud registration is recorded as the third face mesh.

[0228] Step 240, Art Adjustment (Refine):

[0229] After obtaining the third person face mesh, you can make art adjustments, specifically, you can repair the wiring of the third person face mesh, especially the wiring of the eyes, mouth and other structural details in the third person mesh. In addition, you can also beautify the third person face mesh.

[0230] The method provided by the embodiment of the present invention combines global principal component analysis and local principal component analysis to perform three-dimensional face modeling, so as to reduce the error between the second face mesh and the face point cloud, and reduce the mesh deformation when the face point cloud is wrapped based on the second face mesh, thereby greatly reducing the repair time required by the modeler and effectively improving the efficiency of face modeling.

[0231] It should be noted that the face modeling methods provided in the above embodiments can be applied to the generation of virtual digital humans, thereby solving the problems of serious homogeneity of character settings, low quality and efficiency of virtual digital humans.

[0232] The face modeling method provided by each embodiment of the present invention is used to generate virtual digital humans. Compared with manual modeling by a modeler, the generation efficiency of virtual digital humans can be improved. In addition, the virtual digital humans have high-quality face modeling. The hyper-realistic virtual humans obtained in this way can be used for user interaction, virtual digital human videos, and virtual digital human services, so that users can visually have the same experience as interacting with real people, real-life videos, and real-life services.

[0233] Based on any of the above embodiments, Figure 4 Schematic diagram of the structure of the face modeling device provided by the present invention. Figure 4 As shown, the device comprises:

[0234] An initial unit 410 is used to obtain a face point cloud and an initial face mesh, wherein the initial face mesh is obtained based on a global principal component analysis model;

[0235] A global fitting unit 420, configured to perform global fitting on the face point cloud based on the initial face mesh to obtain a first face mesh;

[0236] A local modeling unit 430, configured to perform local principal component analysis modeling on each face region based on the first face mesh to obtain an initial region mesh of each face region;

[0237] The local fitting unit 440 is used to perform local fitting of each face region on the face point cloud based on the initial region mesh of each face region to obtain a second face mesh.

[0238] The device provided by the embodiment of the present invention combines global principal component analysis and local principal component analysis to perform three-dimensional face modeling to reduce the error between the second face mesh and the face point cloud, and reduce the mesh deformation when the face point cloud is wrapped based on the second face mesh, thereby greatly reducing the repair time required by the modeler and effectively improving the efficiency of face modeling.

[0239] Based on any of the above embodiments, the local fitting unit is specifically used for:

[0240] Based on the grid points of the initial regional grids of the facial regions and the nearest neighboring points of the grid points of the initial regional grids in the facial point cloud, the constraint value of the local fitting is determined, and the parameters of the local principal component analysis model corresponding to each initial regional grid are adjusted based on the constraint value of the local fitting until the fitting convergence condition is met, thereby obtaining the second facial grid.

[0241] Based on any of the above embodiments, the local fitting unit is specifically used for:

[0242] Determining a fitting distance constraint value based on the distance between the grid point and the nearest neighbor point;

[0243] Determining a fitting smoothness constraint value based on a difference in smoothness of the grid points before and after parameter adjustment of the local principal component analysis model;

[0244] A constraint value of the local fitting is determined based on the fitting distance constraint value and the fitting smoothness constraint value.

[0245] Based on any of the above embodiments, the local fitting unit is specifically used for:

[0246] Determining a first distance constraint value based on the distance between the vertex in the grid point and the nearest neighbor point;

[0247] Determining a second distance constraint value based on the distance between the facial feature points in the grid points and the facial feature points in the facial point cloud;

[0248] The fitting distance constraint value is determined based on the first distance constraint value and the second distance constraint value.

[0249] Based on any of the above embodiments, the local fitting unit is specifically used for:

[0250] determining a first smoothness of the grid point based on a position of the grid point before a parameter of the local principal component analysis model is adjusted;

[0251] determining a second smoothness of the grid point based on the position of the grid point after the parameters of the local principal component analysis model are adjusted;

[0252] The fitting smoothness constraint value is determined based on a difference between a second smoothness of the grid points and the first smoothness.

[0253] Based on any of the above embodiments, the local fitting unit is specifically used for:

[0254] Determining a junction constraint value based on the consistency of positions of junction grid points in the initial area grids of each face area, wherein the junction grid points are grid points at the junctions of the initial area grids;

[0255] Determining parameter constraint values ​​based on differences between parameters of the local principal component analysis models corresponding to the initial regional grids;

[0256] Determining the axis constraint value of the region based on the positional relationship between the regional central grid point and the coordinate axis, wherein the regional central grid point is a grid point on the central axis of the face in the initial regional grid;

[0257] The constraint value of the local fitting is determined based on at least one of the boundary constraint value, the parameter constraint value, the axis constraint value in the area, the distance constraint value and the fitting smoothness constraint value.

[0258] Based on any of the above embodiments, the device further includes a neck posture correction unit, which is used to:

[0259] Based on the neck posture model of the face point cloud, posture correction is performed on the initial region mesh of each face region to obtain the posture-corrected initial region mesh, and the posture-corrected initial region mesh is used for local fitting.

[0260] Based on any of the above embodiments, the neck posture correction unit is further used for:

[0261] Based on the neck points in the first face mesh, the neck point cloud in the face point cloud is fitted to obtain a neck posture model required for fitting.

[0262] Based on any of the above embodiments, the local fitting unit is specifically used for:

[0263] Based on the initial regional meshes of each face region, locally fit each face region to the face point cloud aligned with the first face mesh to obtain a second face mesh;

[0264] The facial point cloud aligned with the first facial mesh pose is obtained based on the global fitting.

[0265] Based on any of the above embodiments, the device further includes a registration unit, configured to:

[0266] The second face mesh and the face point cloud are registered to obtain a third face mesh.

[0267] Based on any of the above embodiments, the registration unit is specifically used for:

[0268] Based on the grid points of the second face grid and the nearest neighbor points of the grid points of the second face grid in the face point cloud, a registration constraint value is determined, and the positions of the grid points in the second face grid are adjusted based on the registration constraint value until the registration convergence condition is met, so as to obtain the third face grid.

[0269] Based on any of the above embodiments, the registration unit is specifically used for:

[0270] Determining a registration distance constraint value based on the distance between the grid point and the nearest neighbor point;

[0271] Determining a registration smoothness constraint value based on a difference in smoothness of the grid points before and after position adjustment;

[0272] The registration constraint value is determined based on the registration distance constraint value and the registration smoothness constraint value.

[0273] Based on any of the above embodiments, the registration unit is specifically used for:

[0274] determining a rigid deformation constraint value based on a transformation relationship of each face in the second face mesh before and after the positions of the mesh points in the second face mesh are adjusted;

[0275] Determining a mouth closing constraint value based on a difference between an edge formed by upper and lower lip vertices in the grid points before and after position adjustment;

[0276] Determining a second central axis constraint value based on a positional relationship between a second central axis grid point and the coordinate axis, wherein the second central axis grid point is a grid point on the central axis of the face in the second face grid;

[0277] The registration constraint value is determined based on at least one of the rigid deformation constraint value, the mouth closure constraint value, the second central axis constraint value, the registration distance constraint value, and the registration smoothness constraint value.

[0278] Based on any of the above embodiments, the registration unit is specifically used for:

[0279] Registering the second face mesh and the face point cloud aligned with the first face mesh to obtain a third face mesh;

[0280] The facial point cloud aligned with the first facial mesh pose is obtained based on the global fitting.

[0281] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the face modeling method, which includes:

[0282] Obtaining a face point cloud and an initial face mesh, wherein the initial face mesh is obtained based on a global principal component analysis model;

[0283] Based on the initial face mesh, globally fit the face point cloud to obtain a first face mesh;

[0284] Based on the first face grid, perform local principal component analysis modeling on each face region to obtain an initial region grid of each face region;

[0285] Based on the initial regional meshes of each face region, local fitting of each face region is performed on the face point cloud to obtain a second face mesh.

[0286] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a grid device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0287] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the face modeling method provided by the above methods, the method includes:

[0288] Obtaining a face point cloud and an initial face mesh, wherein the initial face mesh is obtained based on a global principal component analysis model;

[0289] Based on the initial face mesh, globally fit the face point cloud to obtain a first face mesh;

[0290] Based on the first face grid, perform local principal component analysis modeling on each face region to obtain an initial region grid of each face region;

[0291] Based on the initial regional meshes of each face region, local fitting of each face region is performed on the face point cloud to obtain a second face mesh.

[0292] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the face modeling method provided by the above methods is implemented, and the method includes:

[0293] Obtaining a face point cloud and an initial face mesh, wherein the initial face mesh is obtained based on a global principal component analysis model;

[0294] Based on the initial face mesh, globally fit the face point cloud to obtain a first face mesh;

[0295] Based on the first face grid, perform local principal component analysis modeling on each face region to obtain an initial region grid of each face region;

[0296] Based on the initial regional meshes of each face region, local fitting of each face region is performed on the face point cloud to obtain a second face mesh.

[0297] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed over multiple grid units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0298] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a grid device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0299] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A face modeling method, characterized in that: include: Obtaining a face point cloud and an initial face mesh, wherein the initial face mesh is obtained based on a global principal component analysis model; Based on the initial face mesh, globally fit the face point cloud to obtain a first face mesh; Based on the first face grid, perform local principal component analysis modeling on each face region to obtain an initial region grid of each face region; Based on the initial regional meshes of the respective face regions, locally fitting the face point clouds for the respective face regions to obtain a second face mesh; The method of performing local fitting of each face region on the face point cloud based on the initial region mesh of each face region to obtain a second face mesh comprises: Based on the grid points of the initial regional grids of the facial regions and the nearest neighboring points of the grid points of the initial regional grids in the facial point cloud, the constraint value of the local fitting is determined, and the parameters of the local principal component analysis model corresponding to each initial regional grid are adjusted based on the constraint value of the local fitting until the fitting convergence condition is met, thereby obtaining the second facial grid.

2. The face modeling method according to claim 1, characterized in that: The determining of the constraint value of the local fitting based on the grid points of the initial regional grid of each face region and the nearest neighboring points of the grid points of the initial regional grid in the face point cloud includes: Determining a fitting distance constraint value based on the distance between the grid point and the nearest neighbor point; Determining a fitting smoothness constraint value based on a difference in smoothness of the grid points before and after parameter adjustment of the local principal component analysis model; A constraint value of the local fitting is determined based on the fitting distance constraint value and the fitting smoothness constraint value.

3. The face modeling method according to claim 2, characterized in that: The step of determining a fitting distance constraint value based on the distance between the grid point and the nearest neighbor point comprises: Determining a first distance constraint value based on the distance between the vertex in the grid point and the nearest neighbor point; Determining a second distance constraint value based on the distance between the facial feature points in the grid points and the facial feature points in the facial point cloud; The fitting distance constraint value is determined based on the first distance constraint value and the second distance constraint value.

4. The face modeling method according to claim 2, characterized in that: The step of determining the fitting smoothness constraint value based on the difference in smoothness of the grid points before and after the parameter adjustment of the local principal component analysis model comprises: determining a first smoothness of the grid point based on a position of the grid point before a parameter of the local principal component analysis model is adjusted; determining a second smoothness of the grid point based on the position of the grid point after the parameters of the local principal component analysis model are adjusted; The fitting smoothness constraint value is determined based on a difference between a second smoothness of the grid points and the first smoothness.

5. The face modeling method according to claim 2, characterized in that: The determining the constraint value of the local fitting based on the fitting distance constraint value and the fitting smoothness constraint value comprises: Determining a junction constraint value based on the consistency of positions of junction grid points in the initial area grids of each face area, wherein the junction grid points are grid points at the junctions of the initial area grids; Determining parameter constraint values ​​based on differences between parameters of the local principal component analysis models corresponding to the initial regional grids; Determining the axis constraint value of the region based on the positional relationship between the regional central grid point and the coordinate axis, wherein the regional central grid point is a grid point on the central axis of the face in the initial regional grid; The constraint value of the local fitting is determined based on at least one of the boundary constraint value, the parameter constraint value, the axis constraint value in the area, the distance constraint value and the fitting smoothness constraint value.

6. The face modeling method according to claim 1, characterized in that: Before the initial regional mesh based on each face region, performing local fitting of each face region on the face point cloud to obtain a second face mesh, the method further includes: Based on the neck posture model of the face point cloud, posture correction is performed on the initial region mesh of each face region to obtain the posture-corrected initial region mesh, and the posture-corrected initial region mesh is used for local fitting.

7. The face modeling method according to claim 6, characterized in that: The step of determining the neck posture model comprises: Based on the neck points in the first face mesh, the neck point cloud in the face point cloud is fitted to obtain a neck posture model required for fitting.

8. The face modeling method according to claim 1, characterized in that: The method of performing local fitting of each face region on the face point cloud based on the initial region mesh of each face region to obtain a second face mesh comprises: Based on the initial regional meshes of each face region, locally fit each face region to the face point cloud aligned with the first face mesh to obtain a second face mesh; The facial point cloud aligned with the first facial mesh pose is obtained based on the global fitting.

9. The face modeling method according to claim 1, characterized in that: Based on the initial regional mesh of each face region, the face point cloud is locally fitted in each face region to obtain a second face mesh, and then the method further includes: The second face mesh and the face point cloud are registered to obtain a third face mesh.

10. The face modeling method according to claim 9, characterized in that: The registering the second face mesh and the face point cloud to obtain a third face mesh comprises: Based on the grid points of the second face grid and the nearest neighbor points of the grid points of the second face grid in the face point cloud, a registration constraint value is determined, and the positions of the grid points in the second face grid are adjusted based on the registration constraint value until the registration convergence condition is met, so as to obtain the third face grid.

11. The face modeling method according to claim 10, characterized in that: The determining of the registration constraint value based on the grid points of the second face grid and the nearest neighbor points of the grid points of the second face grid in the face point cloud comprises: Determining a registration distance constraint value based on the distance between the grid point and the nearest neighbor point; Determining a registration smoothness constraint value based on a difference in smoothness of the grid points before and after position adjustment; The registration constraint value is determined based on the registration distance constraint value and the registration smoothness constraint value.

12. The face modeling method according to claim 11, characterized in that: The step of determining the registration constraint value based on the registration distance constraint value and the registration smoothness constraint value comprises: determining a rigid deformation constraint value based on a transformation relationship of each face in the second face mesh before and after the positions of the mesh points in the second face mesh are adjusted; Determining a mouth closing constraint value based on a difference between an edge formed by upper and lower lip vertices in the grid points before and after position adjustment; Determining a second central axis constraint value based on a positional relationship between a second central axis grid point and the coordinate axis, wherein the second central axis grid point is a grid point on the central axis of the face in the second face grid; The registration constraint value is determined based on at least one of the rigid deformation constraint value, the mouth closure constraint value, the second central axis constraint value, the registration distance constraint value, and the registration smoothness constraint value.

13. The face modeling method according to claim 9, characterized in that: The registering the second face mesh and the face point cloud to obtain a third face mesh comprises: Registering the second face mesh and the face point cloud aligned with the first face mesh to obtain a third face mesh; The facial point cloud aligned with the first facial mesh pose is obtained based on the global fitting.

14. A face modeling device, characterized in that: include: An initial unit, used to obtain a face point cloud and an initial face mesh, wherein the initial face mesh is obtained based on a global principal component analysis model; A global fitting unit, configured to perform global fitting on the face point cloud based on the initial face mesh to obtain a first face mesh; A local modeling unit, configured to perform local principal component analysis modeling on each face region based on the first face grid to obtain an initial region grid of each face region; A local fitting unit, configured to perform local fitting of each face region on the face point cloud based on the initial region mesh of each face region to obtain a second face mesh; The local fitting unit is specifically used for: Based on the grid points of the initial regional grids of the facial regions and the nearest neighboring points of the grid points of the initial regional grids in the facial point cloud, the constraint value of the local fitting is determined, and the parameters of the local principal component analysis model corresponding to each initial regional grid are adjusted based on the constraint value of the local fitting until the fitting convergence condition is met, thereby obtaining the second facial grid.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the face modeling method as described in any one of claims 1 to 13 is implemented.

16. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the face modeling method according to any one of claims 1 to 13 is implemented.

17. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by at least one processor, the face modeling method according to any one of claims 1 to 13 is implemented.

Images