Face Modeling Method, Device, Electronic Device, and Storage Medium

Abstract

The present invention provides a face modeling method, apparatus, electronic device, and storage medium. The method includes: obtaining face images from multiple perspectives; performing feature point marking on the face images from multiple perspectives to obtain the positions of each image feature point from multiple perspectives; based on the positions of each image feature point from multiple perspectives, mapping each image feature point to a face point cloud to obtain each point cloud feature point in the face point cloud; and based on each point cloud feature point, performing point cloud registration on an initial mesh and the face point cloud to obtain a face mesh. The method, apparatus, electronic device, and storage medium provided by the present invention realize fast marking of face feature points by performing feature point marking on the face images from multiple perspectives to obtain the positions of each image feature point from multiple perspectives and mapping the positions of each image feature point from multiple perspectives to the face point cloud to obtain the point cloud feature points on the face point cloud, thereby improving the face modeling efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular, to a face modeling method, apparatus, electronic device, and storage medium. Background Art

[0002] With the development of artificial intelligence technology, three-dimensional (3D) face models have been applied in many scenarios.

[0003] Traditional face modeling includes steps such as three-dimensional point reconstruction, constructing a triangular mesh, and texture mapping. Among them, three-dimensional point reconstruction usually requires a modeler to annotate three-dimensional face feature points on a face point cloud, which takes a lot of time for the modeler to annotate face feature points, resulting in low face modeling efficiency. Summary of the Invention

[0004] The present invention provides a face modeling method, apparatus, electronic device, and storage medium to solve the defect in the prior art that a modeler needs to perform face feature point annotation, which is time-consuming and laborious.

[0005] The present invention provides a face modeling method, including:

[0006] Obtain face images from multiple perspectives;

[0007] Perform feature point marking on the face images from the multiple perspectives to obtain the positions of the respective image feature points in the multiple perspectives;

[0008] Based on the positions of the respective image feature points in the multiple perspectives, map the respective image feature points to a face point cloud to obtain respective point cloud feature points in the face point cloud;

[0009] Based on the respective point cloud feature points, perform point cloud registration on an initial mesh and the face point cloud to obtain a face mesh.

[0010] According to the face modeling method provided by the present invention, the step of mapping the respective image feature points to the face point cloud based on the positions of the respective image feature points in the multiple perspectives to obtain the respective point cloud feature points in the face point cloud includes:

[0011] Based on the positions of the respective image feature points in the multiple perspectives, map the respective image feature points to the face point cloud to obtain candidate positions of the respective point cloud feature points on the face point cloud in the multiple perspectives;

[0012] Based on the candidate positions of the respective point cloud feature points in the multiple perspectives, locate the respective point cloud feature points in the face point cloud.

[0013] According to the face modeling method provided by the present invention, the face image is obtained by camera shooting;

[0014] Mapping the respective image feature points to the face point cloud based on the positions of the respective image feature points at multiple perspectives includes:

[0015] Mapping the respective image feature points to the face point cloud based on the shooting parameters of the camera at multiple perspectives and the positions of the respective image feature points at multiple perspectives.

[0016] According to a face modeling method provided by the present invention, the face image is obtained by rasterizing the face point cloud;

[0017] Mapping the respective image feature points to the face point cloud based on the positions of the respective image feature points at multiple perspectives includes:

[0018] Mapping the respective image feature points to the face point cloud based on the rasterization parameters of the face point cloud at multiple perspectives and the positions of the respective image feature points at multiple perspectives.

[0019] According to a face modeling method provided by the present invention, locating the point cloud feature points on the face point cloud based on the candidate positions of the respective point cloud feature points at multiple perspectives includes:

[0020] Performing weighted fusion on the candidate positions of the respective point cloud feature points at multiple perspectives based on the visual weights of the respective point cloud feature points at each perspective to obtain the positions of the respective point cloud feature points in the face point cloud.

[0021] According to a face modeling method provided by the present invention, performing point cloud registration on the initial mesh and the face point cloud based on the respective point cloud feature points to obtain a face mesh includes:

[0022] Performing point cloud registration on the initial mesh and the face point cloud based on the point cloud feature points and the face regions to which the respective sampling points in the face point cloud belong to obtain a face mesh.

[0023] The present invention also provides a face modeling method, including:

[0024] Obtaining a face point cloud;

[0025] Performing face region segmentation on the face point cloud to obtain the face regions to which the respective sampling points in the face point cloud belong;

[0026] Performing point cloud registration on the initial mesh and the face point cloud based on the respective point cloud feature points in the face point cloud and the face regions to which the respective sampling points in the face point cloud belong to obtain a face mesh.

[0027] A face modeling method provided by the present invention, which performs point cloud registration on an initial mesh and the face point cloud based on each point cloud feature point in the face point cloud and the face region to which each sampling point in the face point cloud belongs, obtaining a face mesh, includes:

[0028] Performing registration on the initial mesh and the face point cloud based on forward point pairs and / or reverse point pairs, and each mesh feature point in the initial mesh and each point cloud feature point in the face point cloud, to obtain the face mesh;

[0029] The forward point pair includes a mesh point in the initial mesh and the forward nearest neighbor point of the mesh point in the face point cloud, and the mesh point and the forward nearest neighbor point belong to the same face region;

[0030] The reverse point pair includes the sampling point and the reverse nearest neighbor point of the sampling point in the initial mesh, and the sampling point and the reverse nearest neighbor point belong to the same face region.

[0031] A face modeling method provided by the present invention, in the case of performing registration on the initial mesh and the face point cloud based on forward point pairs and reverse point pairs, and each mesh feature point in the initial mesh and each point cloud feature point in the face point cloud, the face region segmentation of the face point cloud to obtain the face region to which each sampling point in the face point cloud belongs, includes:

[0032] Performing face region segmentation on the face point cloud based on a first segmentation threshold to obtain a first face region to which each sampling point in the face point cloud belongs,

[0033] Performing face region segmentation on the face point cloud based on a second segmentation threshold to obtain a second face region to which each sampling point in the face point cloud belongs;

[0034] The first segmentation threshold is less than the second segmentation threshold;

[0035] The forward point pair is determined based on the first face region, and the reverse point pair is determined based on the second face region.

[0036] A face modeling method provided by the present invention, which performs registration on the initial mesh and the face point cloud based on forward point pairs and / or reverse point pairs, and each mesh feature point in the initial mesh and each point cloud feature point in the face point cloud, to obtain the face mesh, includes:

[0037] Determining a forward constraint based on the distance between the mesh point and the forward nearest neighbor point in the forward point pair, and / or determining a reverse constraint based on the distance between the sampling point and the reverse nearest neighbor point in the reverse point pair;

[0038] Determine feature point constraints based on the distances between the respective grid feature points and the corresponding point cloud feature points;

[0039] Determine registration constraints based on the forward constraints and / or the reverse constraints, and the feature point constraints;

[0040] Register the initial grid and the face point cloud based on the registration constraints to obtain the face grid.

[0041] According to a face modeling method provided by the present invention, the forward nearest neighbor points are determined based on the distances between the grid points and the sampling points within the same face area, and the normal directions of the grid points and the normal directions of the sampling points within the same face area;

[0042] The reverse nearest neighbor points are determined based on the distances between the sampling points and the grid points within the same face area, and the normal directions of the sampling points and the normal directions of the grid points within the same face area.

[0043] According to a face modeling method provided by the present invention, the step of performing face area segmentation on the face point cloud to obtain the face areas to which the respective sampling points in the face point cloud belong includes:

[0044] Obtain two-dimensional images of the face point cloud from multiple perspectives;

[0045] Perform face area segmentation on the two-dimensional images from the multiple perspectives respectively to obtain the area segmentation results of the two-dimensional images from the multiple perspectives;

[0046] Map the area segmentation results of the two-dimensional images from the multiple perspectives to the face point cloud to obtain the candidate areas to which the respective sampling points in the face point cloud belong from multiple perspectives;

[0047] Based on the candidate areas to which the respective sampling points belong from multiple perspectives, obtain the face areas to which the respective sampling points belong.

[0048] According to a face modeling method provided by the present invention, the face areas include a facial area, a neck area, and an ear area.

[0049] The present invention also provides a face modeling device, including:

[0050] An image acquisition unit for acquiring face images from multiple perspectives;

[0051] An image marking unit for marking feature points of the face images from the multiple perspectives to obtain the positions of the respective image feature points from multiple perspectives;

[0052] A feature point mapping unit, configured to map each of the image feature points to a face point cloud based on the positions of the image feature points in multiple perspectives, so as to obtain each point cloud feature point in the face point cloud;

[0053] A registration unit, configured to perform point cloud registration on the initial mesh and the face point cloud based on each of the point cloud feature points, so as to obtain a face mesh.

[0054] The present invention further provides a face modeling device, including:

[0055] A point cloud acquisition unit, configured to acquire a face point cloud;

[0056] A region segmentation unit, configured to perform face region segmentation on the face point cloud to obtain the face region to which each sampling point in the face point cloud belongs;

[0057] A region registration unit, configured to perform point cloud registration on the initial mesh and the face point cloud based on each of the point cloud feature points in the face point cloud and the face region to which each sampling point in the face point cloud belongs, so as to obtain a face mesh.

[0058] The present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, the face modeling method described in any one of the above is implemented.

[0059] The present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the face modeling method described in any one of the above is implemented.

[0060] The present invention further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the face modeling method described in any one of the above is implemented.

[0061] The face modeling method, device, electronic device, and storage medium provided by the present invention realize fast marking of face feature points by marking feature points of face images in multiple perspectives to obtain the positions of each image feature point in multiple perspectives, and mapping the positions of each image feature point in multiple perspectives to the face point cloud to obtain point cloud feature points on the face point cloud, thereby improving the face modeling efficiency. Description of the Drawings

[0062] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0063] Figure 1 It is one of the flow diagrams of the face modeling method provided by the present invention;

[0064] Figure 2 It is the flow diagram of the feature point mapping method provided by the present invention;

[0065] Figure 3 It is the second flow diagram of the face modeling method provided by the present invention;

[0066] Figure 4 It is the flow diagram of the face area segmentation method of the face point cloud provided by the present invention;

[0067] Figure 5 It is the third flow diagram of the face modeling method provided by the present invention;

[0068] Figure 6 It is one of the structural diagrams of the face modeling device provided by the present invention;

[0069] Figure 7 It is the second structural diagram of the face modeling device provided by the present invention;

[0070] Figure 8 It is the structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0071] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0072] Three-dimensional face modeling is a difficult problem in the field of three-dimensional modeling. It often requires modelers to construct according to the face contour and facial feature layout, etc., which takes a lot of time. Among them, for the annotation of face feature points, it takes a modeler a lot of time to complete. How to improve the efficiency of face feature point annotation and thus improve the three-dimensional face modeling efficiency is still an urgent problem to be solved.

[0073] To address this problem, an embodiment of the present invention provides a face modeling method. Figure 1 It is the flow diagram of the face modeling method provided by the present invention. As Figure 1 shown, the method includes:

[0074] Step 110, obtaining face images from multiple perspectives.

[0075] Here, the face image is a two-dimensional image containing a face. The face images from multiple perspectives contain the faces of the same person from multiple perspectives. The face contained in the face image here can be a neutral face, and the expression of the person in the neutral face is a neutral expression, specifically the expression in the natural state where the person closes their mouth naturally, has no expression, and looks straight ahead. Here, one perspective can correspond to one face image or multiple face images. Multiple perspectives can cover the entire face. For example, multiple perspectives can be the front, left side, and right side perspectives. Another example is that 90 face images of the same person from different perspectives can be obtained, and the face images with a face angle greater than 45° are deleted, and the remaining face images with a face angle within 45° are used as the face images for face modeling.

[0076] The face image can be an image directly obtained by taking a photo with a camera, or an image obtained by rendering one or more perspectives selected from a face point cloud. Among them, the face point cloud is the point cloud data obtained by scanning the face of the person to be modeled using devices such as 3D scanners and depth cameras.

[0077] Step 120: Mark the feature points of the face images from the multiple perspectives to obtain the positions of the respective image feature points from the multiple perspectives.

[0078] Specifically, for each face image, feature point marking can be performed separately, thereby obtaining the positions of the image feature points in each face image. It can be understood that the image feature points here are the face feature points in the image dimension, and the positions of the image feature points can be represented as two-dimensional coordinates.

[0079] After obtaining the positions of the image feature points in each face image, position statistics can be performed with the image feature points as the unit, thereby obtaining the positions of the respective image feature points in each face image. And, using the correspondence between the face image and the perspective, the positions of the respective image feature points from the multiple perspectives can be obtained. Here, the positions of the respective image feature points from the multiple perspectives are the positions of the respective image feature points in the face images from the multiple perspectives.

[0080] Here, the marking of feature points for a face image can be achieved through a pre-trained face feature point marking model. The face feature point marking model here is used to mark image feature points for a face image. The training method of the face feature point model may include the following steps: First, collect sample face images and mark the positions of each image feature point in each sample face image. Then, use the sample face images as training samples and the positions of each image feature point in the sample face images as position labels to perform supervised training on the initial model, thereby obtaining the face feature point marking model. The initial model here can be a neural network model with any structure, and the embodiments of the present invention do not make specific limitations on this.

[0081] Step 130, based on the positions of the respective image feature points at multiple perspectives, map the respective image feature points to the face point cloud to obtain each point cloud feature point in the face point cloud.

[0082] Here, the face point cloud and the face images at the above multiple perspectives belong to the same person. The face point cloud is a geometric data representing the shape of a face, which consists of a series of three-dimensional coordinate points, and these points describe the shape, size, position and other features of the face of this person.

[0083] After obtaining the positions of the respective image feature points at multiple perspectives, based on the positions of the image feature points at multiple perspectives, the image feature points can be mapped from the two-dimensional space where the face image is located to the three-dimensional space where the face point cloud is located, thereby obtaining the point cloud feature points on the face point cloud. Here, the point cloud feature points are the three-dimensional face feature points on the face point cloud, and the position of the point cloud feature points can be represented as three-dimensional coordinates.

[0084] Here, mapping the image feature points to point cloud feature points can be achieved through a pre-set mapping relationship. The mapping relationship here can be based on the parameters of the camera used to capture the face image, or can be based on the rendering parameters used to render the face image from the face point cloud. The embodiments of the present invention do not make specific limitations on this.

[0085] It can be understood that compared with directly marking face feature points on a three-dimensional face point cloud, it is easier to mark feature points on a two-dimensional face image, and the acquisition cost of the sample face images required to train the model for marking feature points on the face image is lower and more convenient. Therefore, the reliability and accuracy of marking feature points on a two-dimensional face image are better. After obtaining the positions of the respective image feature points at multiple perspectives, mapping the image feature points to the three-dimensional face point cloud can achieve the marking of feature points for the face point cloud, greatly improving the efficiency of marking feature points for the face point cloud on the premise of ensuring the reliability and accuracy of marking feature points for the face point cloud.

[0086] Step 140: Based on the respective point cloud feature points, perform point cloud registration on the initial mesh and the face point cloud to obtain a face mesh.

[0087] Specifically, after implementing the marking of feature points for the face point cloud, that is, after locating the point cloud feature points in the face point cloud, the point cloud feature points can be used as the three-dimensional face feature points of the face point cloud and applied to the point cloud registration of the initial mesh and the face point cloud. Thus, the initial mesh after point cloud registration, that is, the face mesh, is obtained.

[0088] Here, the initial mesh is the initial face mesh template. The initial mesh can be obtained by performing global principal component analysis (GlobalPCA) on multiple face meshes with different lengths and genders. Alternatively, the initial mesh can also be the face mesh template after point cloud fitting for the face point cloud based on a general face mesh model. The embodiments of the present invention do not make specific limitations on this.

[0089] Furthermore, the method of registering the initial mesh and the face point cloud can be to use the face feature points in the initial mesh and the point cloud feature points in the face point cloud as constraints to adjust the positions of each mesh point in the initial mesh; or it can be to find the nearest neighbor points of the mesh points in the initial mesh in the face point cloud, and use the mesh points and the nearest neighbor points, as well as the face feature points in the initial mesh and the point cloud feature points in the face point cloud as constraints to adjust the positions of each mesh point in the initial mesh, so that the initial mesh can fit the face point cloud better, thereby obtaining the face mesh.

[0090] Optionally, in step 140, the respective point cloud feature points output in step 130 can be directly applied to point cloud registration, or the respective point cloud feature points output in step 130 can be imported into 3D modeling software, such as wrap. The modeler can adjust the respective point cloud feature points in the 3D modeling software, and then apply the adjusted respective point cloud feature points to point cloud registration.

[0091] The method provided by the embodiments of the present invention realizes the rapid marking of face feature points by marking feature points of face images from multiple perspectives, obtaining the positions of each image feature point from multiple perspectives, and mapping the positions of each image feature point from multiple perspectives to the face point cloud, thereby improving the face modeling efficiency.

[0092] Based on the above embodiments, Figure 2 is a schematic flowchart of the feature point mapping method provided by the present invention. As Figure 2 shown, step 130 includes:

[0093] Step 131: Based on the positions of the respective image feature points at multiple perspectives, map the respective image feature points to the face point cloud to obtain the candidate positions of each point cloud feature point on the face point cloud at multiple perspectives.

[0094] Specifically, in the process of mapping the image feature points to the face point cloud, the feature point mapping can be performed perspective by perspective. That is, based on the coordinate transformation relationship from one perspective mapped to the face point cloud, the positions of the respective image feature points in the face image at this perspective are transformed to the face point cloud, that is, the candidate positions of the respective point cloud feature points corresponding to the respective image feature points in the face point cloud at this perspective are obtained.

[0095] It can be understood that the candidate positions of each point cloud feature point here correspond to the perspectives. That is, in the feature point mapping for each perspective, the candidate positions of each point cloud feature point at this perspective can be obtained.

[0096] On this basis, statistics can be performed in units of the point cloud feature points, so as to obtain, for each point cloud feature point, the candidate positions of this point cloud feature point obtained at different perspectives. Here, the number of candidate positions possessed by a point cloud feature point is the same as the number of perspectives, and the candidate positions of a point cloud feature point at different perspectives may be the same or different.

[0097] Step 132: Based on the candidate positions of the respective point cloud feature points at multiple perspectives, locate each point cloud feature point in the face point cloud.

[0098] Specifically, after obtaining the candidate positions of each point cloud feature point at multiple perspectives, the candidate positions of this point cloud feature point at multiple perspectives can be fused in units of the point cloud feature point to determine the position of this point cloud feature point in the face point cloud, that is, to achieve the marked positioning of this point cloud feature point in the face point cloud.

[0099] Here, for any point cloud feature point, the average of the candidate positions of this point cloud feature point at multiple perspectives can be taken as the position of this point cloud feature point in the face point cloud, or the positions of this point cloud feature point at multiple perspectives can be weighted and summed according to the importance of each perspective to obtain the position of this point cloud feature point in the face point cloud. The embodiments of the present invention do not make specific limitations on this.

[0100] Based on any of the above embodiments, the face image is obtained by camera shooting.

[0101] Correspondingly, in Step 131, based on the positions of the respective image feature points at multiple perspectives, mapping the respective image feature points to the face point cloud includes:

[0102] Based on the shooting parameters of the camera at multiple perspectives and the positions of the respective image feature points at multiple perspectives, map the respective image feature points to the face point cloud.

[0103] Specifically, on the premise that face images at multiple perspectives are obtained by multi-perspective shooting of the camera, the parameters corresponding to the multi-perspective shooting of the camera can be obtained, that is, the shooting parameters of the camera at multiple perspectives. It can be understood that the shooting parameters at different perspectives may be different.

[0104] Here, the shooting parameters may include internal parameters and external parameters. Among them, the internal parameters refer to the parameters of the camera itself, such as focal length, pixel size, etc., and the internal parameters can be obtained during the camera calibration process. The external parameters refer to the position and orientation of the camera in the three-dimensional world coordinate system, such as often represented as a rotation matrix and a translation vector, and the external parameters can be calculated from the corresponding points of multiple images collected by the camera.

[0105] After obtaining the shooting parameters of the camera at multiple perspectives, based on the shooting parameters at each perspective, the coordinate transformation relationship from the two-dimensional coordinate system to the three-dimensional coordinate system at each perspective can be determined. On this basis, the positions of the respective image feature points at multiple perspectives can be position-transformed through the coordinate transformation relationships at each perspective, so as to obtain the candidate positions of the corresponding point cloud feature points of the respective image feature points at multiple perspectives.

[0106] Based on any of the above embodiments, the face image is obtained by rasterizing the face point cloud;

[0107] Correspondingly, in step 131, the mapping of the respective image feature points to the face point cloud based on the positions of the respective image feature points at multiple perspectives includes:

[0108] Based on the rasterization parameters of the face point cloud at multiple perspectives and the positions of the respective image feature points at multiple perspectives, map the respective image feature points to the face point cloud.

[0109] Specifically, on the premise that the face point cloud is rasterized at multiple perspectives to obtain face images at multiple perspectives, the rasterization parameters for rasterizing the face point cloud at multiple perspectives can be obtained. Here, the rasterization parameters of one perspective can reflect the correspondence between the sampling points in the face point cloud and the pixel points in the face image at this perspective.

[0110] Thus, the positions of the respective image feature points at multiple perspectives can be position-transformed through the rasterization parameters at multiple perspectives, so as to obtain the candidate positions of the corresponding point cloud feature points of the respective image feature points at multiple perspectives.

[0111] Taking the position transformation of any image feature point from any perspective as an example, the rasterization parameters from this perspective reflect the corresponding relationship between the sampling points in the face point cloud and the pixel points in the face image from this perspective. The position of the image feature point from this perspective is the position of the pixel point corresponding to the image feature point in the face image from this perspective. Based on the rasterization parameters from this perspective, the sampling point corresponding to the pixel point in the face point cloud can be determined, and the position of the sampling point is the candidate position of the point cloud feature point corresponding to the image feature point from this perspective.

[0112] Based on any of the above embodiments, in step 132, the positioning of the point cloud feature points on the face point cloud based on the candidate positions of the point cloud feature points from multiple perspectives includes:

[0113] Based on the visual weights of the point cloud feature points from each perspective, the candidate positions of the point cloud feature points from multiple perspectives are weighted and fused to obtain the positions of the point cloud feature points in the face point cloud.

[0114] Specifically, the image feature points included in the face images from different perspectives may be different. For example, in the left face image, the feature points on the right face are invisible, and in the right face image, the feature points on the left face are invisible. Therefore, for each perspective, the visual weights of the point cloud feature points from each perspective can be set.

[0115] Here, for any point cloud feature point from any perspective, the visual weight reflects whether the point cloud feature point is visible from this perspective. Further, the visual weight can reflect the visible probability of the point cloud feature point from this perspective. And for the convenience of calculation, for a point cloud feature point, the sum of the visual weights of the point cloud feature point from each perspective can be 1, and from any perspective, the closer the position of the point cloud feature point corresponding to the face image from this perspective is to the center, the higher the visual weight of the point cloud feature point from this perspective.

[0116] After determining the visual weights of the point cloud feature points from each perspective, the candidate positions of the point cloud feature points from multiple perspectives can be weighted and fused. Taking any point cloud feature point as an example, the candidate positions of the point cloud feature point from each perspective can be weighted and summed with the visual weights of the point cloud feature point from each perspective, and the result of the weighted sum is the position of the point cloud feature point in the face point cloud.

[0117] The method provided by the embodiments of the present invention realizes the fusion of the candidate positions of the point cloud feature points from each perspective by setting the visual weights of the point cloud feature points from each perspective, thereby improving the marking efficiency of the face point cloud feature points while ensuring the accuracy of the positioning of the point cloud feature points in the face point cloud.

[0118] Based on any of the above embodiments, step 140, performing point cloud registration on the initial mesh and the face point cloud based on the respective point cloud feature points to obtain a face mesh, includes:

[0119] Performing point cloud registration on the initial mesh and the face point cloud based on the point cloud feature points and the face regions to which the respective sampling points in the face point cloud belong, to obtain a face mesh.

[0120] Specifically, currently during point cloud registration, the registration effect for face regions such as ears and necks is usually relatively poor. The reason is that during the point cloud registration process, it is impossible to distinguish the face regions to which the respective sampling points in the face point cloud belong, resulting in chaotic registration between the grid points in the initial mesh and the sampling points in the face point cloud and affecting the registration effect.

[0121] For any of these problems, in the embodiments of the present invention, the face regions to which the respective sampling points in the face point cloud belong are used together as reference factors for point cloud registration of the initial mesh and the face point cloud. Thereby, during the point cloud registration process, not only can registration be performed based on the grid feature points in the initial mesh and the point cloud feature points in the face point cloud, but also registration can be performed between each grid point in the initial mesh and the nearest neighbor points in the face point cloud that belong to the same face region, thus avoiding the situation where the registration effect is affected due to confusion of face regions during registration.

[0122] Exemplarily, in some examples, the specific steps of step 140 are the same as steps 310 to 330 described later. For specific details, refer to the following text and no further elaboration will be provided here. Of course, it can be understood that in other embodiments, other methods can also be used to perform step 140 and achieve point cloud registration, and the examples in this embodiment do not impose undue limitations on this application.

[0123] The method provided by the embodiments of the present invention combines point cloud feature points and the face regions to which the respective sampling points in the face point cloud belong to perform point cloud registration of the initial mesh and the face point cloud, thereby ensuring the registration effect of the face mesh, especially the registration effect of the face mesh in regions such as ears and necks.

[0124] Based on any of the above embodiments, Figure 3 is the second flow diagram of the face modeling method provided by the present invention. As Figure 3 shown, the method includes:

[0125] Step 310, obtaining a face point cloud.

[0126] Here, the face point cloud is the face point cloud data of the person to be modeled. The point cloud data can be obtained through face scanning, or can be obtained through pictures including the face of the person to be modeled. The face point cloud includes point cloud feature points. The face point cloud is a kind of geometric data representing the face shape, which consists of a series of three-dimensional coordinate points, and these points describe the features such as the shape, size and position of the face. The face point cloud can be obtained in various ways, such as being collected by devices such as 3D scanners and depth cameras. Exemplarily, the face point cloud here can be obtained through the foregoing steps 110-130. Of course, it can also be obtained through other means. The examples in this embodiment do not impose inappropriate restrictions on this application.

[0127] Step 320, perform face region segmentation on the face point cloud to obtain the face regions to which the respective sampling points in the face point cloud belong.

[0128] Specifically, performing face region segmentation on the face point cloud means determining the face region to which each sampling point in the face point cloud belongs. The face region segmentation here can be a process of classifying the respective sampling points in the face point cloud into respective face regions. For example, the face point cloud can be directly input into a pre-trained neural grid model for face region segmentation of the face point cloud, and the neural network model realizes face region segmentation for the face point cloud; or, face region segmentation can be performed on face images of multiple viewpoints corresponding to the face point cloud, and the segmentation result is mapped to the three-dimensional space where the face point cloud is located to obtain the face region segmentation result of the face point cloud, that is, to obtain the face regions to which the respective sampling points in the face point cloud belong. The embodiments of the present invention do not make specific limitations on this.

[0129] The face region segmentation referred to here can be to divide the face point cloud into a facial region, a neck region and an ear region, or can be divided according to the facial features and the neck. The embodiments of the present invention do not make specific limitations on this.

[0130] Step 330, based on the respective point cloud feature points in the face point cloud and the face regions to which the respective sampling points in the face point cloud belong, perform point cloud registration on the initial mesh and the face point cloud to obtain a face mesh.

[0131] Specifically, the respective point cloud feature points in the face point cloud, that is, the three-dimensional face feature points on the face point cloud, and the position of the point cloud feature points can be represented as three-dimensional coordinates. The point cloud feature points can be marked by a modeler, or image feature points can be marked on face images of multiple viewpoints corresponding to the face point cloud, and the image feature points are mapped to the three-dimensional space where the face point cloud is located, so as to obtain the respective point cloud feature points in the face point cloud. The embodiments of the present invention do not make specific limitations on this.

[0132] The initial grid is the initial face grid template, which can be obtained by performing global principal component analysis on multiple face grids with different lengths and genders. Alternatively, the initial grid can also be the face grid template after point cloud fitting for the face point cloud based on a general face grid model. The embodiments of the present invention do not make specific limitations on this.

[0133] After obtaining the respective point cloud feature points in the face point cloud and the face regions to which the respective sampling points in the face point cloud belong, point cloud registration for the initial grid and the face point cloud can be performed. During the point cloud registration process, not only can registration be performed based on the grid feature points in the initial grid and the point cloud feature points in the face point cloud, but also registration can be performed between each grid point in the initial grid and the nearest neighbor point in the face point cloud that belongs to the same face region, thereby avoiding the situation where the registration effect is affected due to confusion of face regions during registration.

[0134] The registration result obtained therefrom, that is, the grid after adjusting the positions of each grid point in the initial grid, is denoted herein as the face grid.

[0135] The method provided by the embodiments of the present invention combines the point cloud feature points and the face regions to which the respective sampling points in the face point cloud belong to perform point cloud registration for the initial grid and the face point cloud, thereby ensuring the registration effect of the face grid, especially the registration effect of the face grid in regions such as the ears and neck.

[0136] Based on any of the above embodiments, in step 330, the performing point cloud registration for the initial grid and the face point cloud based on the respective point cloud feature points in the face point cloud and the face regions to which the respective sampling points in the face point cloud belong to obtain a face grid includes:

[0137] Performing registration for the initial grid and the face point cloud based on the forward point pairs and the respective grid feature points in the initial grid and the respective point cloud feature points in the face point cloud to obtain the face grid;

[0138] The forward point pairs include the grid points in the initial grid and the forward nearest neighbor points of the grid points in the face point cloud, and the grid points and the forward nearest neighbor points belong to the same face region.

[0139] Specifically, during the process of performing registration for the initial grid and the face point cloud, not only can the respective grid feature points in the initial grid and the respective point cloud feature points in the face point cloud be referred to, but also the forward point pairs constructed from the initial grid and the face point cloud can be referred to.

[0140] Among them, the grid feature points are the face feature points in the initial grid, the grid feature points can be pre-marked, and the grid feature points correspond one-to-one with the point cloud feature points in the face point cloud.

[0141] The forward point pair includes the grid points in the initial grid and the nearest neighbor points found in the face point cloud that belong to the same face region as the grid points. That is, in the process of determining the forward point pair, it is necessary to apply the face region to which the grid points in the initial grid belong and the face regions to which the sampling points in the face point cloud belong. Thus, in the process of determining the forward point pair, the sampling points that belong to the same face region as the grid points can be first screened out from the face point cloud, and then the nearest neighbor point of the grid points can be found from the sampling points that belong to the same face region as the grid points. The nearest neighbor point thus determined is denoted as the forward nearest neighbor point of the grid points. For example, the grid points in the ear region can only find the forward nearest neighbor points from the sampling points that belong to the ear region in the face point cloud. It can be understood that the forward here is the process of fitting the grid to the point cloud.

[0142] Thus, the application of the forward point pair can constrain the registration of grid points and sampling points within the same face region, thereby avoiding the situation where the registration effect is affected by the confusion of face regions during registration.

[0143] The method provided by the embodiments of the present invention performs registration on the initial grid and the face point cloud based on the forward point pair and the grid feature points in the initial grid and the point cloud feature points in the face point cloud. Among them, the forward point pair can constrain the registration of grid points and sampling points to be carried out within the same face region, thereby avoiding the situation where the registration effect is affected by the confusion of face regions during registration, ensuring the reliability of point cloud registration, reducing the workload of subsequent manual adjustment, and thus improving the reliability and efficiency of face modeling.

[0144] Based on any of the above embodiments, in step 330, the point cloud registration of the initial grid and the face point cloud is performed based on the point cloud feature points in the face point cloud and the face regions to which the sampling points in the face point cloud belong, and a face grid is obtained, including:

[0145] Performing registration on the initial grid and the face point cloud based on the reverse point pair and the grid feature points in the initial grid and the point cloud feature points in the face point cloud to obtain the face grid;

[0146] The reverse point pair includes the sampling points and the reverse nearest neighbor points of the sampling points in the initial grid, and the sampling points and the reverse nearest neighbor points belong to the same face region.

[0147] Specifically, in the process of registering the initial grid and the face point cloud, not only can the grid feature points in the initial grid and the point cloud feature points in the face point cloud be referred to, but also the reverse point pair constructed by the initial grid and the face point cloud can be referred to.

[0148] Among them, the grid feature points are the face feature points in the initial grid. The grid feature points can be pre-marked and are in one-to-one correspondence with the point cloud feature points in the face point cloud.

[0149] The reverse point pair includes the sampling points in the face point cloud and the nearest neighbor points found in the initial grid that belong to the same face area as the sampling point. That is, in the process of determining the reverse point pair, it is necessary to apply the face area to which the grid points in the initial grid belong and the face area to which each sampling point in the face point cloud belongs. Thus, in the process of determining the reverse point pair, first filter out the grid points in the initial grid that belong to the same face area as the sampling point, and then find the nearest neighbor point of the sampling point among the grid points that belong to the same face area as the sampling point. The nearest neighbor point thus determined is denoted as the reverse nearest neighbor point of the sampling point. For example, the sampling points in the ear area can only find the reverse nearest neighbor points from the grid points in the initial grid that belong to the ear area. It can be understood that the reverse here is the process of fitting the point cloud to the grid.

[0150] Therefore, the application of the reverse point pair can constrain the registration of grid points and sampling points within the same face area, thereby avoiding the situation where the registration effect is affected by the confusion of face areas during registration.

[0151] The method provided by the embodiments of the present invention performs registration on the initial grid and the face point cloud based on the reverse point pair and each grid feature point in the initial grid and each point cloud feature point in the face point cloud. Among them, the reverse point pair can constrain the registration of grid points and sampling points to be carried out within the same face area, thereby avoiding the situation where the registration effect is affected by the confusion of face areas during registration, ensuring the reliability of point cloud registration, reducing the workload of subsequent manual adjustment, and thus improving the reliability and efficiency of face modeling.

[0152] Based on any of the above embodiments, in step 330, the performing point cloud registration on the initial grid and the face point cloud based on each point cloud feature point in the face point cloud and the face area to which each sampling point in the face point cloud belongs to obtain a face grid includes:

[0153] Performing registration on the initial grid and the face point cloud based on the forward point pair and the reverse point pair, and each grid feature point in the initial grid and each point cloud feature point in the face point cloud to obtain the face grid;

[0154] The forward point pair includes the grid points in the initial grid and the forward nearest neighbor points of the grid points in the face point cloud, and the grid points and the forward nearest neighbor points belong to the same face area;

[0155] The reverse point pair includes the sampling point and the reverse nearest neighbor point of the sampling point in the initial mesh, and the sampling point and the reverse nearest neighbor point belong to the same face region.

[0156] Specifically, in the process of registering the initial mesh and the face point cloud, not only can the grid feature points in the initial mesh and the point cloud feature points in the face point cloud be referred to, but also the forward point pairs and reverse point pairs constructed from the initial mesh and the face point cloud can be referred to.

[0157] Among them, the grid feature points are the face feature points in the initial mesh. The grid feature points can be pre-labeled, and the grid feature points and the point cloud feature points in the face point cloud are in one-to-one correspondence.

[0158] The forward point pair includes the grid point in the initial mesh and the nearest neighbor point found in the face point cloud that belongs to the same face region as the grid point. That is, in the process of determining the forward point pair, the face region to which the grid point in the initial mesh belongs and the face regions to which the sampling points in the face point cloud belong need to be applied. Thus, in the process of determining the forward point pair, the sampling points that belong to the same face region as the grid point can be first screened out from the face point cloud, and then the nearest neighbor point of the grid point can be found from the sampling points that belong to the same face region as the grid point. The nearest neighbor point thus determined is denoted as the forward nearest neighbor point of the grid point. For example, for the grid points in the ear region, the forward nearest neighbor point can only be found from the sampling points in the face point cloud that belong to the ear region. It can be understood that the forward here is the process of fitting the grid to the point cloud.

[0159] The reverse point pair includes the sampling point in the face point cloud and the nearest neighbor point found in the initial mesh that belongs to the same face region as the sampling point. That is, in the process of determining the reverse point pair, the face region to which the grid point in the initial mesh belongs and the face regions to which the sampling points in the face point cloud belong need to be applied. Thus, in the process of determining the reverse point pair, the grid points that belong to the same face region as the sampling point can be first screened out from the initial mesh, and then the nearest neighbor point of the sampling point can be found from the grid points that belong to the same face region as the sampling point. The nearest neighbor point thus determined is denoted as the reverse nearest neighbor point of the sampling point. For example, for the sampling points in the ear region, the reverse nearest neighbor point can only be found from the grid points in the initial mesh that belong to the ear region. It can be understood that the reverse here is the process of fitting the point cloud to the grid.

[0160] Thus, whether it is a forward point pair or a reverse point pair, the registration of grid points and sampling points is constrained within the same face region, thereby avoiding the situation that the registration effect is affected due to the confusion of face regions during registration.

[0161] The method provided by the embodiment of the present invention performs registration on the initial mesh and the face point cloud based on the forward point pairs and reverse point pairs, and the grid feature points in the initial mesh and the point cloud feature points in the face point cloud. Among them, the forward point pairs and reverse point pairs can constrain the registration of grid points and sampling points in the same face area, thereby avoiding the situation that the registration effect is affected by the confusion of face areas during registration, ensuring the reliability of point cloud registration, reducing the workload of subsequent manual adjustment, and thus improving the reliability and efficiency of face modeling.

[0162] Based on any of the above embodiments, in the case of applying both forward point pairs and reverse point pairs in point cloud registration, in step 320, the face area segmentation of the face point cloud to obtain the face areas to which the respective sampling points in the face point cloud belong includes:

[0163] Performing face area segmentation on the face point cloud based on a first segmentation threshold to obtain a first face area to which each sampling point in the face point cloud belongs, and the forward point pairs are determined based on the first face area. And, performing face area segmentation on the face point cloud based on a second segmentation threshold to obtain a second face area to which each sampling point in the face point cloud belongs, and the reverse point pairs are determined based on the second face area. The first segmentation threshold is less than the second segmentation threshold.

[0164] It can be understood that here the embodiment of the present invention takes the case where both forward point pairs and reverse point pairs are set as an example for illustration. However, in other embodiments, it may also be the case where forward point pairs are set and reverse point pairs are not set. At this time, face area segmentation can be performed on the face point cloud based on the first segmentation threshold to obtain the first face area to which each sampling point in the face point cloud belongs, and the forward point pairs can be determined; or, it may also be the case where reverse point pairs are set and forward point pairs are not set. At this time, face area segmentation can be performed on the face point cloud based on the second segmentation threshold to obtain the second face area to which each sampling point in the face point cloud belongs, and the reverse point pairs can be determined.

[0165] Specifically, face area segmentation for the face point cloud can be achieved based on the segmentation threshold.

[0166] Here, the application of the segmentation threshold in face area segmentation can specifically be to directly input the face point cloud into a neural grid model that has been pre-trained for face area segmentation of the face point cloud, and the neural network model realizes face area segmentation for the face point cloud to obtain the probability that each sampling point in the face point cloud belongs to each face area. Furthermore, the face area to which each sampling point belongs can be determined by comparing the segmentation threshold and the probability.

[0167] Alternatively, face region segmentation can be performed on face images from multiple perspectives corresponding to the face point cloud, and the segmentation results can be mapped to the three-dimensional space where the face point cloud is located. Since the segmentation results mapped from different perspectives to the face point cloud may be different, that is, for each sampling point in the face point cloud, there can be multiple face regions it belongs to under multiple perspectives. Based on this, the score of each sampling point belonging to the face region can be determined, and the score can be compared with the segmentation threshold to determine the face region to which each sampling point belongs.

[0168] It can be seen that the segmentation threshold reflects the strictness of face region segmentation in face region segmentation. It can be understood that the larger the value of the segmentation threshold, the stricter the requirements for face region segmentation, and the more accurate the segmentation result. The smaller the value of the segmentation threshold, the looser the requirements for face region segmentation, and the more comprehensive the attribution of each sampling point in the segmentation result, and it is less likely to have sampling points that cannot be attributed to any face region.

[0169] Therefore, two segmentation thresholds can be determined, namely the first segmentation threshold and the second segmentation threshold, and two segmentation results for the face point cloud can be obtained therefrom. It can be understood that the first segmentation threshold is less than the second segmentation threshold. The face region to which each sampling point in the face point cloud belongs based on the first segmentation threshold is denoted as the first face region, and the face region to which each sampling point in the face point cloud belongs based on the second segmentation threshold is denoted as the second face region.

[0170] It can be understood that the delimitation of the first face region is more comprehensive compared to the second face region; the delimitation of the second face region is more accurate compared to the first face region.

[0171] During the point cloud registration process, the first face region obtained by partitioning can be applied to determine the forward point pairs, so that in the process of finding the forward point pairs, each grid point can find the forward nearest neighbor point in the face point cloud to ensure the comprehensiveness of the construction of the forward point pairs, and further guarantee the point cloud registration effect.

[0172] In addition, the second face region obtained by partitioning can be applied to determine the reverse point pairs, so that in the process of finding the reverse point pairs, the sampling points belonging to each second face region can find the reverse nearest neighbor point in the corresponding face region of the initial grid to ensure the accuracy of the construction of the reverse point pairs, and further guarantee the point cloud registration effect.

[0173] Based on any of the above embodiments, in step 330, the registering the initial grid and the face point cloud to obtain the face grid based on the forward point pairs and / or reverse point pairs, and each grid feature point in the initial grid and each point cloud feature point in the face point cloud includes:

[0174] Determine a forward constraint based on the distance between the grid points and the forward nearest neighbor points in the forward point pair, and / or determine a reverse constraint based on the distance between the sampled points and the reverse nearest neighbor points in the reverse point pair;

[0175] Determine a feature point constraint based on the distance between each grid feature point and the corresponding point cloud feature point;

[0176] Determine a registration constraint based on the forward constraint and / or the reverse constraint, and the feature point constraint;

[0177] Register the initial grid and the face point cloud based on the registration constraint to obtain the face grid.

[0178] Specifically, the registration of the initial grid and the face point cloud can be achieved based on the registration constraint. That is, the positions of the grid points in the initial grid can be adjusted based on the registration constraint. After the position adjustment, the forward point pair and / or the reverse point pair can be re-determined, and a new registration constraint can be calculated based on this, and the positions of the grid points in the initial grid can be adjusted again until the registration convergence condition is met.

[0179] The registration convergence condition here can be that the registration constraint is less than or equal to a preset threshold, or the number of position adjustments in the point cloud registration reaches the target number. The embodiments of the present invention do not make specific limitations on this.

[0180] Here, the calculation of the registration constraint can be achieved by calculating the forward constraint and / or the reverse constraint, and calculating the feature point constraint.

[0181] It can be understood that in the solution of registering the initial grid and the face point cloud based on the forward point pair, and each grid feature point in the initial grid and each point cloud feature point in the face point cloud, the registration constraint can be achieved by calculating the forward constraint and calculating the feature point constraint; in the solution of registering the initial grid and the face point cloud based on the reverse point pair, and each grid feature point in the initial grid and each point cloud feature point in the face point cloud, the registration constraint can be achieved by calculating the reverse constraint and calculating the feature point constraint; in the solution of registering the initial grid and the face point cloud based on the forward point pair, the reverse point pair, and each grid feature point in the initial grid and each point cloud feature point in the face point cloud, the registration constraint can be achieved by calculating the forward constraint, the reverse constraint, and calculating the feature point constraint.

[0182] Among them, the forward constraint is determined based on the distance between the grid point in the forward point pair and the forward nearest neighbor point. Specifically, the distance between the grid point in each forward point pair and the forward nearest neighbor point can be calculated, and the distances between the grid points and the forward nearest neighbor points in each forward point pair are accumulated as the forward constraint. It can be understood that the farther the distance between the grid point and the forward nearest neighbor point, the greater the forward constraint; the closer the distance between the grid point and the forward nearest neighbor point, the smaller the forward constraint.

[0183] The reverse constraint is determined based on the distance between the sampling point in the reverse point pair and the reverse nearest neighbor point. Specifically, the distance between the sampling point in each reverse point pair and the reverse nearest neighbor point can be calculated, and the distances between the sampling points and the reverse nearest neighbor points in each reverse point pair are accumulated as the reverse constraint. It can be understood that the farther the distance between the sampling point and the reverse nearest neighbor point, the greater the reverse constraint; the closer the distance between the sampling point and the reverse nearest neighbor point, the smaller the reverse constraint.

[0184] The feature point constraint is determined based on the distance between each grid feature point and the corresponding point cloud feature point. Specifically, the distance between each network feature point and the corresponding point cloud feature point can be calculated, and the calculated distances are accumulated as the feature point constraint. It can be understood that the farther the distance between the grid feature point and the corresponding point cloud feature point, the greater the feature point constraint; the closer the distance between the grid feature point and the corresponding point cloud feature point, the smaller the feature point constraint.

[0185] After obtaining the forward constraint and / or reverse constraint, and calculating the feature point constraint, the forward constraint and / or reverse constraint, and the calculated feature point constraint can be directly added as the registration constraint, or the forward constraint and / or reverse constraint, and the calculated feature point constraint can be weighted and summed as the registration constraint. The embodiments of the present invention do not make specific limitations on this.

[0186] Based on any of the above embodiments, in the forward point pair applied in step 330, the forward nearest neighbor point is determined based on the distance between the grid point and each sampling point in the same face area, and the normal direction of the grid point and the normal directions of each sampling point in the same face area;

[0187] In the reverse point pair, the reverse nearest neighbor point is determined based on the distance between the sampling point and each grid point in the same face area, and the normal direction of the sampling point and the normal directions of each grid point in the same face area.

[0188] Specifically, during the process of point cloud registration, when finding the nearest neighbor points, including finding the forward nearest neighbor points and / or the backward nearest neighbor points, in addition to applying the coordinates of the grid points in the initial grid and the sampling points in the face point cloud under the same face area to determine the distance between the grid points and the sampling points, the normal direction of the grid points on the initial grid and the normal direction of the sampling points on the face point cloud can also be used as the features applied in the nearest neighbor point search.

[0189] Here, the normal direction refers to the normal direction of the plane tangent to the surface of the initial grid or the face point cloud at that point. Through the normal direction, the orientation of the surface of the initial grid or the face point cloud can be judged.

[0190] Using the normal direction of the grid points on the initial grid and the normal direction of the sampling points on the face point cloud as the features applied in the nearest neighbor point search can further improve the reliability of the nearest neighbor point search. Thus, for areas such as the ears with more surface undulations and stronger normal direction features, determining the nearest neighbor points in this way can greatly improve the registration effect.

[0191] Based on any of the above embodiments, Figure 4 is a schematic flowchart of the face area segmentation method for the face point cloud provided by the present invention. As Figure 4 shown, step 320 includes:

[0192] Step 321, obtaining two-dimensional images of the face point cloud from multiple perspectives.

[0193] Here, the two-dimensional images from multiple perspectives corresponding to the face point cloud, that is, the face images from multiple perspectives corresponding to the same person as the face point cloud. The two-dimensional images here contain a face. And one perspective can correspond to one two-dimensional image or multiple two-dimensional images. Multiple perspectives can cover the complete face. For example, multiple perspectives can be three perspectives: the front, the left side, and the right side. Another example is that 90 face images of the same person from different perspectives can be obtained, and the face images with a face angle greater than 45° are deleted, and the remaining two-dimensional images with a face angle within 45° are used as the images for face area segmentation of the face point cloud.

[0194] Here, the two-dimensional image can be an image directly obtained by camera shooting, or an image rendered by selecting one or more perspectives from the face point cloud.

[0195] Step 322, respectively performing face area segmentation on the two-dimensional images from multiple perspectives to obtain the area segmentation results of the two-dimensional images from multiple perspectives.

[0196] Specifically, for each obtained two-dimensional image, face region segmentation can be performed separately to obtain the region segmentation results of each two-dimensional image. Here, the face region segmentation for the two-dimensional image can be achieved through a pre-trained face segmentation model. The face segmentation model here is used to perform face region segmentation on the two-dimensional image. The training method of the face segmentation model can include the following steps: First, collect sample face images and label the positions of each face region in each sample face image. Then, use the sample face images as training samples and the positions of each face region in the sample face images as position labels to perform supervised training on the initial model, thereby obtaining a face region segmentation model. The initial model here can be a neural network model with any structure, and the embodiments of the present invention do not make specific limitations on this.

[0197] Step 323: Map the region segmentation results of the two-dimensional images at the multiple viewpoints to the face point cloud to obtain the candidate regions to which each sampling point in the face point cloud belongs at the multiple viewpoints.

[0198] Specifically, after obtaining the region segmentation results of the two-dimensional regions at each viewpoint, the region segmentation results can be mapped from the two-dimensional space to the three-dimensional space where the face point cloud is located, thereby achieving face region segmentation for the face point cloud.

[0199] In this process, considering that the region segmentation results at different viewpoints may be the same or different, the region segmentation results at each viewpoint can be mapped separately, so as to obtain the face regions to which each sampling point in the face point cloud belongs at each viewpoint. Here, it is denoted as the candidate regions to which each sampling point belongs at each viewpoint.

[0200] Step 324: Based on the candidate regions to which each sampling point belongs at the multiple viewpoints, obtain the face region to which each sampling point belongs.

[0201] Specifically, for the candidate regions to which each sampling point belongs at the multiple viewpoints, the number of viewpoints to which each sampling point belongs to each candidate region can be counted. For example, a sampling point may belong to the neck region at 5 viewpoints, the face region at 14 viewpoints, and the ear region at 0 viewpoints. Based on this, the candidate region corresponding to the largest number of viewpoints can be selected as the face region to which the sampling point belongs. Or, the number of viewpoints of each candidate region can be compared with a preset segmentation threshold. If there is a candidate region whose number of viewpoints is greater than or equal to the segmentation threshold, then this candidate region is used as the face region to which the sampling point belongs. If the number of viewpoints of each candidate region is less than the segmentation threshold, then this sampling point is determined as a sampling point outside the region, that is, this sampling point does not belong to any face region.

[0202] It can be understood that by setting a larger segmentation threshold, a more accurate face region segmentation result can be obtained; by setting a smaller segmentation threshold, sampling points belonging to each face region can be found as comprehensively as possible in the face point cloud. The setting of the segmentation threshold can be determined based on the actual situation.

[0203] Based on any of the above embodiments, the face region includes a facial region, a neck region, and an ear region.

[0204] Specifically, the face point cloud can be divided into three regions: a facial region, a neck region, and an ear region. In this way, during the point cloud registration process, registration can be performed separately for the face, neck, and ears to avoid the situation where the confusion between the face, neck, and ears affects the registration effect.

[0205] Based on any of the above embodiments, Figure 5 is the third flow schematic diagram of the face modeling method provided by the present invention. As Figure 5 shown, the method includes:

[0206] First, obtain the face point cloud of the same person, as well as face images from multiple perspectives.

[0207] Secondly, perform feature point marking on the face images from multiple perspectives to obtain the positions of each image feature point in multiple perspectives. Subsequently, based on the positions of each image feature point in multiple perspectives, map each image feature point to the face point cloud to obtain each point cloud feature point in the face point cloud.

[0208] In addition, perform face region segmentation on the face images from multiple perspectives respectively to obtain the region segmentation results of the face images from multiple perspectives. Subsequently, map the region segmentation results of the face images from multiple perspectives to the face point cloud to obtain the face regions to which each sampling point in the face point cloud belongs.

[0209] Then, combine each point cloud feature point in the face point cloud and the face regions to which each sampling point in the face point cloud belongs, and perform point cloud registration on the initial mesh and the face point cloud. During this process, each point cloud feature point in the face point cloud performs self-iteration.

[0210] After completing the point cloud registration, the modeler can then adjust each point cloud feature point in the face point cloud. Based on the adjusted point cloud feature points, point cloud registration can be continued, and finally, the face mesh obtained by point cloud registration can be obtained.

[0211] It can be understood that through the self-iteration of each point cloud feature point during the point cloud registration process, the accuracy of the point cloud feature points is improved, and the modeler only needs to make fine adjustments to a small number of point cloud feature points. Thus, the time required for face modeling can be greatly reduced.

[0212] It should be noted that the face modeling methods provided in the above embodiments can be applied to the generation of virtual digital humans, thereby solving problems such as serious homogenization of character settings, low quality and efficiency in virtual digital humans.

[0213] Applying the face modeling methods provided in the embodiments of the present invention to the generation of virtual digital humans can improve the generation efficiency of virtual digital humans compared to manual modeling by modelers. Moreover, virtual digital humans have high-quality facial modeling. The resulting hyper-realistic virtual humans can be used for user interaction, virtual digital human videos, and virtual digital human services, etc., so that users can have an experience similar to interacting with real people, real person videos, and real person services visually.

[0214] Based on any of the above embodiments, Figure 6 is one of the schematic structural diagrams of the face modeling device provided by the present invention, as Figure 6 shown, the device includes:

[0215] An image acquisition unit, configured to acquire face images from multiple perspectives;

[0216] An image marking unit, configured to mark feature points on the face images from the multiple perspectives to obtain the positions of each image feature point from the multiple perspectives;

[0217] A feature point mapping unit, configured to map each image feature point to a face point cloud based on the positions of each image feature point from the multiple perspectives to obtain each point cloud feature point in the face point cloud;

[0218] A registration unit, configured to perform point cloud registration on an initial mesh and the face point cloud based on each point cloud feature point to obtain a face mesh.

[0219] The device provided in the embodiments of the present invention realizes fast marking of face feature points by marking feature points on face images from multiple perspectives to obtain the positions of each image feature point from the multiple perspectives and mapping the positions of each image feature point from the multiple perspectives to the face point cloud to obtain point cloud feature points on the face point cloud, thereby improving the face modeling efficiency.

[0220] Based on any of the above embodiments, the feature point mapping unit is specifically configured to:

[0221] Map each image feature point to a face point cloud based on the positions of each image feature point from the multiple perspectives to obtain candidate positions of each point cloud feature point on the face point cloud from the multiple perspectives;

[0222] Locate each point cloud feature point in the face point cloud based on the candidate positions of each point cloud feature point from the multiple perspectives.

[0223] Based on any of the above embodiments, the face image is obtained by camera shooting;

[0224] The feature point mapping unit is specifically configured to:

[0225] Based on the shooting parameters of the camera at multiple perspectives and the positions of the respective image feature points at multiple perspectives, map the respective image feature points to the face point cloud.

[0226] Based on any of the above embodiments, the face image is obtained by rasterizing the face point cloud;

[0227] The feature point mapping unit is specifically configured to:

[0228] Based on the rasterization parameters of the face point cloud at multiple perspectives and the positions of the respective image feature points at multiple perspectives, map the respective image feature points to the face point cloud.

[0229] Based on any of the above embodiments, the feature point mapping unit is specifically configured to:

[0230] Based on the visible weights of the respective point cloud feature points at each perspective, perform weighted fusion on the candidate positions of the respective point cloud feature points at multiple perspectives to obtain the positions of the respective point cloud feature points in the face point cloud.

[0231] Based on any of the above embodiments, the registration unit is specifically configured to:

[0232] Based on the point cloud feature points and the face regions to which the respective sampling points in the face point cloud belong, perform point cloud registration on the initial mesh and the face point cloud to obtain a face mesh.

[0233] Figure 7 is the second structural schematic diagram of the face modeling device provided by the present invention, as Figure 7 shown, the device includes:

[0234] A point cloud acquisition unit 710, configured to acquire a face point cloud;

[0235] A region segmentation unit 720, configured to perform face region segmentation on the face point cloud to obtain the face regions to which the respective sampling points in the face point cloud belong;

[0236] A region registration unit 730, configured to perform point cloud registration on the initial mesh and the face point cloud based on the respective point cloud feature points in the face point cloud and the face regions to which the respective sampling points in the face point cloud belong, to obtain a face mesh.

[0237] The device provided by the embodiment of the present invention combines the feature points of the point cloud and the face regions to which the sampling points in the face point cloud belong, and performs point cloud registration on the initial mesh and the face point cloud, thereby ensuring the registration effect of the face mesh, especially the registration effect of the face mesh in regions such as ears and necks.

[0238] Based on any of the above embodiments, the region registration unit is specifically configured to:

[0239] Based on the forward point pairs and / or reverse point pairs, and the grid feature points in the initial mesh and the point cloud feature points in the face point cloud, register the initial mesh and the face point cloud to obtain the face mesh;

[0240] The forward point pairs include the grid points in the initial mesh and the forward nearest neighbor points of the grid points in the face point cloud, and the grid points and the forward nearest neighbor points belong to the same face region;

[0241] The reverse point pairs include the sampling points and the reverse nearest neighbor points of the sampling points in the initial mesh, and the sampling points and the reverse nearest neighbor points belong to the same face region.

[0242] Based on any of the above embodiments, the region segmentation unit is specifically configured to:

[0243] Perform face region segmentation on the face point cloud based on a first segmentation threshold to obtain the first face regions to which the sampling points in the face point cloud belong,

[0244] and / or,

[0245] Perform face region segmentation on the face point cloud based on a second segmentation threshold to obtain the second face regions to which the sampling points in the face point cloud belong;

[0246] The first segmentation threshold is less than the second segmentation threshold;

[0247] The forward point pairs are determined based on the first face regions, and the reverse point pairs are determined based on the second face regions.

[0248] Based on any of the above embodiments, the region registration unit is specifically configured to:

[0249] Determine a forward constraint based on the distance between the grid points and the forward nearest neighbor points in the forward point pairs;

[0250] Determine a reverse constraint based on the distance between the sampling points and the reverse nearest neighbor points in the reverse point pairs;

[0251] Determine a feature point constraint based on the distance between the grid feature points and the corresponding point cloud feature points;

[0252] Determine the registration constraint based on the forward constraint and / or the reverse constraint, and the feature point constraint;

[0253] Register the initial mesh and the face point cloud based on the registration constraint to obtain the face mesh.

[0254] Based on any of the above embodiments, the forward nearest neighbor point is determined based on the distance between the grid point and each sampling point in the same face area, and the normal direction of the grid point and the normal direction of each sampling point in the same face area;

[0255] The reverse nearest neighbor point is determined based on the distance between the sampling point and each grid point in the same face area, and the normal direction of the sampling point and the normal direction of each grid point in the same face area.

[0256] Based on any of the above embodiments, the region segmentation unit is specifically configured to:

[0257] Obtain two-dimensional images of the face point cloud from multiple perspectives;

[0258] Perform face region segmentation on the two-dimensional images from multiple perspectives respectively to obtain the region segmentation results of the two-dimensional images from multiple perspectives;

[0259] Map the region segmentation results of the two-dimensional images from multiple perspectives to the face point cloud to obtain the candidate regions to which each sampling point in the face point cloud belongs from multiple perspectives;

[0260] Based on the candidate regions to which each sampling point belongs from multiple perspectives, obtain the face region to which each sampling point belongs.

[0261] Based on any of the above embodiments, the face region includes a facial region, a neck region, and an ear region.

[0262] Figure 8 Illustrates a schematic physical structure diagram of an electronic device, as Figure 8 shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the face modeling method, and the method includes:

[0263] Obtain face images from multiple perspectives;

[0264] Mark feature points on the face images under the multiple perspectives to obtain the positions of the respective image feature points under the multiple perspectives;

[0265] Based on the positions of the respective image feature points under the multiple perspectives, map the respective image feature points to the face point cloud to obtain respective point cloud feature points in the face point cloud;

[0266] Based on the respective point cloud feature points, perform point cloud registration on the initial mesh and the face point cloud to obtain a face mesh.

[0267] Or, it includes:

[0268] Obtain a face point cloud;

[0269] Perform face region segmentation on the face point cloud to obtain the face regions to which the respective sampling points in the face point cloud belong;

[0270] Based on the respective point cloud feature points in the face point cloud and the face regions to which the respective sampling points in the face point cloud belong, perform point cloud registration on the initial mesh and the face point cloud to obtain a face mesh.

[0271] In addition, when the logical instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0272] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the face modeling method provided by the above-mentioned various methods. The method includes:

[0273] Obtain face images under multiple perspectives;

[0274] Mark feature points on the face images under the multiple perspectives to obtain the positions of the respective image feature points under the multiple perspectives;

[0275] Based on the positions of the respective image feature points at multiple viewpoints, map the respective image feature points to the face point cloud to obtain respective point cloud feature points in the face point cloud;

[0276] Based on the respective point cloud feature points, perform point cloud registration on the initial mesh and the face point cloud to obtain a face mesh.

[0277] Or, it includes:

[0278] Obtain a face point cloud;

[0279] Perform face region segmentation on the face point cloud to obtain the face regions to which the respective sampling points in the face point cloud belong;

[0280] Based on the respective point cloud feature points in the face point cloud and the face regions to which the respective sampling points in the face point cloud belong, perform point cloud registration on the initial mesh and the face point cloud to obtain a face mesh.

[0281] On the other hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is configured to execute the face modeling method provided by the above-mentioned respective methods. The method includes:

[0282] Obtain face images at multiple viewpoints;

[0283] Perform feature point marking on the face images at multiple viewpoints to obtain the positions of the respective image feature points at multiple viewpoints;

[0284] Based on the positions of the respective image feature points at multiple viewpoints, map the respective image feature points to the face point cloud to obtain the respective point cloud feature points in the face point cloud;

[0285] Based on the respective point cloud feature points, perform point cloud registration on the initial mesh and the face point cloud to obtain a face mesh.

[0286] Or, it includes:

[0287] Obtain a face point cloud;

[0288] Perform face region segmentation on the face point cloud to obtain the face regions to which the respective sampling points in the face point cloud belong;

[0289] Based on the respective point cloud feature points in the face point cloud and the face regions to which the respective sampling points in the face point cloud belong, perform point cloud registration on the initial mesh and the face point cloud to obtain a face mesh.

[0290] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0291] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0292] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A face modeling method, characterized in that: include: Acquire facial images from multiple perspectives; Marking feature points of the face images under the multiple viewing angles to obtain positions of feature points of each image under the multiple viewing angles; Based on the positions of each image feature point under multiple viewing angles, each image feature point is mapped to a face point cloud to obtain each point cloud feature point in the face point cloud; Based on the feature points of each point cloud, performing point cloud registration on the initial grid and the face point cloud to obtain a face grid; The initial grid is an initial face grid template, and the point cloud registration of the initial grid and the face point cloud based on the point cloud feature points includes: Adjusting the position of each grid point in the initial grid based on the relationship between the facial feature points in the initial grid and the point cloud feature points in the facial point cloud; The mapping of each image feature point to a face point cloud based on the position of each image feature point under multiple viewing angles to obtain each point cloud feature point in the face point cloud includes: Based on the positions of each image feature point at multiple viewing angles, each image feature point is mapped to a face point cloud to obtain candidate positions of each point cloud feature point on the face point cloud at multiple viewing angles; Based on the visible weights of the point cloud feature points at each viewing angle, weighted fusion is performed on the candidate positions of the point cloud feature points at multiple viewing angles to obtain the positions of the point cloud feature points in the face point cloud; The visible weight reflects the visibility probability of the point cloud feature point at the corresponding viewing angle. The closer the position of the point cloud feature point in the face image at a certain viewing angle is to the center, the higher the visible weight of the point cloud feature point at the viewing angle is.

2. The face modeling method according to claim 1, characterized in that: The facial image is captured by a camera; The mapping of each image feature point to a face point cloud based on the position of each image feature point under multiple viewing angles includes: Based on the shooting parameters of the camera at multiple viewing angles and the positions of the image feature points at multiple viewing angles, the image feature points are mapped to a face point cloud.

3. The face modeling method according to claim 1, characterized in that: The face image is obtained by rasterizing the face point cloud; The mapping of each image feature point to a face point cloud based on the position of each image feature point under multiple viewing angles includes: Based on the rasterization parameters of the face point cloud at multiple viewing angles and the positions of the image feature points at multiple viewing angles, the image feature points are mapped to the face point cloud.

4. The face modeling method according to any one of claims 1 to 3, characterized in that: The step of performing point cloud registration on the initial mesh and the face point cloud based on the feature points of each point cloud to obtain a face mesh comprises: Based on the point cloud feature points and the face area to which each sampling point in the face point cloud belongs, point cloud registration is performed on the initial grid and the face point cloud to obtain a face grid.

5. A face modeling method, characterized in that: include: Get face point cloud; Performing face region segmentation on the face point cloud to obtain the face region to which each sampling point in the face point cloud belongs; Based on each point cloud feature point in the face point cloud and the face area to which each sampling point in the face point cloud belongs, performing point cloud registration on the initial grid and the face point cloud to obtain a face grid; The initial grid is an initial face grid template, and the point cloud registration of the initial grid and the face point cloud based on each point cloud feature point in the face point cloud and the face area to which each sampling point in the face point cloud belongs includes: Adjusting the position of each grid point in the initial grid based on the relationship between the facial feature points in the initial grid and the point cloud feature points in the facial point cloud, and the relationship between each grid point in the initial grid and the nearest neighbor points belonging to the same facial region in the facial point cloud; Each point cloud feature point in the face point cloud is obtained by the following method: Acquire facial images from multiple perspectives; Marking feature points of the face images under the multiple viewing angles to obtain positions of feature points of each image under the multiple viewing angles; Based on the positions of each image feature point at multiple viewing angles, each image feature point is mapped to a face point cloud to obtain candidate positions of each point cloud feature point on the face point cloud at multiple viewing angles; Based on the visible weights of the point cloud feature points at each viewing angle, weighted fusion is performed on the candidate positions of the point cloud feature points at multiple viewing angles to obtain the positions of the point cloud feature points in the face point cloud; The visible weight reflects the visibility probability of the point cloud feature point at the corresponding viewing angle. The closer the position of the point cloud feature point in the face image at a certain viewing angle is to the center, the higher the visible weight of the point cloud feature point at the viewing angle is.

6. The face modeling method according to claim 5, characterized in that: The step of performing point cloud registration on the initial grid and the face point cloud based on each point cloud feature point in the face point cloud and the face region to which each sampling point in the face point cloud belongs to obtain a face grid includes: Based on the forward point pairs and / or the reverse point pairs, and each grid feature point in the initial grid and each point cloud feature point in the face point cloud, the initial grid and the face point cloud are registered to obtain the face grid; The positive point pair includes a grid point in the initial grid and a positive nearest neighbor point of the grid point in the face point cloud, and the grid point and the positive nearest neighbor point belong to the same face region; The reverse point pair includes the sampling point and the reverse nearest neighbor point of the sampling point in the initial grid, and the sampling point and the reverse nearest neighbor point belong to the same face region.

7. The face modeling method according to claim 6, characterized in that: In the case where the initial grid and the face point cloud are registered based on the forward point pairs and the reverse point pairs, and each grid feature point in the initial grid and each point cloud feature point in the face point cloud, The performing face region segmentation on the face point cloud to obtain the face region to which each sampling point in the face point cloud belongs includes: Performing face region segmentation on the face point cloud based on a first segmentation threshold to obtain a first face region to which each sampling point in the face point cloud belongs, Performing face region segmentation on the face point cloud based on a second segmentation threshold to obtain a second face region to which each sampling point in the face point cloud belongs; The first segmentation threshold is less than the second segmentation threshold; The forward point pair is determined based on the first face region, and the reverse point pair is determined based on the second face region.

8. The face modeling method according to claim 6, characterized in that: Based on the forward point pairs and / or the reverse point pairs, and each grid feature point in the initial grid and each point cloud feature point in the face point cloud, the initial grid and the face point cloud are registered to obtain the face grid, including: Determine a forward constraint based on a distance between a grid point in the forward point pair and a forward nearest neighbor point, and / or determine a reverse constraint based on a distance between a sampling point in the reverse point pair and a reverse nearest neighbor point; Determining feature point constraints based on the distance between each grid feature point and the corresponding point cloud feature point; Determining a registration constraint based on the forward constraint and / or the reverse constraint and the feature point constraint; Based on the registration constraint, the initial mesh and the face point cloud are registered to obtain the face mesh.

9. The face modeling method according to claim 6, characterized in that: The forward nearest neighbor point is determined based on the distance between the grid point and each sampling point in the same face region, and the normal direction of the grid point and the normal direction of each sampling point in the same face region; The reverse nearest neighbor point is determined based on the distance between the sampling point and each grid point in the same face area, and the normal direction of the sampling point and the normal direction of each grid point in the same face area.

10. The face modeling method according to claim 5, characterized in that: The performing face region segmentation on the face point cloud to obtain the face region to which each sampling point in the face point cloud belongs includes: Acquire two-dimensional images under multiple viewing angles corresponding to the face point cloud; Performing face region segmentation on the two-dimensional images under the multiple viewing angles respectively to obtain region segmentation results of the two-dimensional images under the multiple viewing angles; Mapping the region segmentation results of the two-dimensional image under the multiple viewing angles to the face point cloud to obtain candidate regions to which each sampling point in the face point cloud belongs under the multiple viewing angles; Based on the candidate regions to which the sampling points belong under multiple viewing angles, the face regions to which the sampling points belong are obtained.

11. The face modeling method according to any one of claims 5 to 10, characterized in that: The face area includes a facial area, a neck area and an ear area.

12. A face modeling device, characterized in that: include: An image acquisition unit, used to acquire facial images from multiple viewing angles; An image marking unit, used to mark feature points of the face images under the multiple viewing angles to obtain the position of each image feature point under the multiple viewing angles; A feature point mapping unit, used to map each image feature point to a face point cloud based on the position of each image feature point at multiple viewing angles, to obtain each point cloud feature point in the face point cloud; A registration unit, configured to perform point cloud registration on the initial grid and the face point cloud based on the feature points of each point cloud to obtain a face grid; The initial mesh is an initial face mesh template, and the registration unit is specifically used for: Adjusting the position of each grid point in the initial grid based on the relationship between the facial feature points in the initial grid and the point cloud feature points in the facial point cloud; The mapping of each image feature point to a face point cloud based on the position of each image feature point under multiple viewing angles to obtain each point cloud feature point in the face point cloud includes: Based on the positions of each image feature point at multiple viewing angles, each image feature point is mapped to a face point cloud to obtain candidate positions of each point cloud feature point on the face point cloud at multiple viewing angles; Based on the visible weights of the point cloud feature points at each viewing angle, weighted fusion is performed on the candidate positions of the point cloud feature points at multiple viewing angles to obtain the positions of the point cloud feature points in the face point cloud; The visible weight reflects the visibility probability of the point cloud feature point at the corresponding viewing angle. The closer the position of the point cloud feature point in the face image at a certain viewing angle is to the center, the higher the visible weight of the point cloud feature point at the viewing angle is.

13. A face modeling device, characterized in that: include: A point cloud acquisition unit, used to acquire a face point cloud; A region segmentation unit, used to perform face region segmentation on the face point cloud to obtain the face region to which each sampling point in the face point cloud belongs; A regional registration unit, configured to perform point cloud registration on an initial grid and the face point cloud based on each point cloud feature point in the face point cloud and a face region to which each sampling point in the face point cloud belongs, so as to obtain a face grid; The initial grid is an initial face grid template, and the region registration unit is specifically used for: Adjusting the position of each grid point in the initial grid based on the relationship between the facial feature points in the initial grid and the point cloud feature points in the facial point cloud, and the relationship between each grid point in the initial grid and the nearest neighbor points belonging to the same facial region in the facial point cloud; Each point cloud feature point in the face point cloud is obtained by the following method: Acquire facial images from multiple perspectives; Marking feature points of the face images under the multiple viewing angles to obtain positions of feature points of each image under the multiple viewing angles; Based on the positions of each image feature point at multiple viewing angles, each image feature point is mapped to a face point cloud to obtain candidate positions of each point cloud feature point on the face point cloud at multiple viewing angles; Based on the visible weights of the point cloud feature points at each viewing angle, weighted fusion is performed on the candidate positions of the point cloud feature points at multiple viewing angles to obtain the positions of the point cloud feature points in the face point cloud; The visible weight reflects the visibility probability of the point cloud feature point at the corresponding viewing angle. The closer the position of the point cloud feature point in the face image at a certain viewing angle is to the center, the higher the visible weight of the point cloud feature point at the viewing angle is.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the face modeling method as described in any one of claims 1 to 11 is implemented.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the face modeling method as claimed in any one of claims 1 to 11 is implemented.

16. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by at least one processor, the face modeling method according to any one of claims 1 to 11 is implemented.

Images