Mesh refinement method and device based on texture credibility and partition normal displacement

Abstract

The application relates to the technical field of three-dimensional face scanning, and discloses a mesh refinement method and device based on texture credibility and partition normal displacement, which comprises the following steps: performing credibility decomposition on a texture image of a scanned face collected by a plurality of cameras to obtain decomposition results of the cameras; constructing an algorithm according to the decomposition results of all the cameras and a plurality of influence data to generate a composite texture credibility map; performing face partition on an initial face mesh of the scanned face obtained according to the composite texture credibility map and semantic region information to obtain a face partition result; performing normal displacement optimization on a to-be-updated mesh vertex determined from the face partition result along a normal based on a target function determined in advance to update the initial face mesh and obtain a backup face mesh; and performing a back projection verification operation on the backup face mesh to determine a refined mesh satisfying a preset refinement condition, so that the accuracy of face mesh refinement is improved.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional facial scanning technology, and in particular to a mesh refinement method and apparatus based on texture reliability and partitioned normal displacement. Background Technology

[0002] Currently, 3D facial scanning systems can typically acquire facial depth information and texture images, and use high-frequency information such as brightness changes, color changes, or texture gradients in the texture images to refine the initial 3D facial mesh reconstructed based on facial depth information, thereby improving the detail expression capability of the facial mesh.

[0003] However, practice has shown that existing facial mesh refinement methods rely solely on high-frequency texture information. The geometric reliability of this high-frequency texture information is easily affected by non-geometric factors such as lighting variations, highlights, shadows, occlusion, hair, blurring, or imaging noise, making accurate judgment difficult. This can lead to non-geometric appearance details being misused in mesh refinement, resulting in low accuracy and potentially causing problems such as abnormal local deformation and distorted organ boundaries. Therefore, proposing a new facial mesh refinement scheme to improve accuracy is crucial. Summary of the Invention This invention provides a mesh refinement method and apparatus based on texture reliability and partitioned normal displacement, which can help improve the accuracy of facial mesh refinement.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a mesh refinement method based on texture reliability and partition normal displacement, the method comprising: The credibility decomposition is performed on the texture image of the scanned face acquired by each of the multiple cameras to obtain the decomposition result of that camera. The decomposition result includes at least one of the base reflection layer, high frequency detail layer, residual layer and anomalous response layer. Based on the decomposition results of all the cameras and the pre-determined multiple influence data construction algorithm, a composite texture confidence map is generated; Based on the composite texture confidence map and semantic region information, the initial facial mesh of the scanned face is partitioned to obtain a facial partitioning result. The partitioning result includes at least one of a refinement region, a boundary-restricted region, a preservation region, and an anomaly shielding region. For the vertices of the mesh to be updated determined from the facial partitioning results, the displacement of the vertices of the mesh to be updated is optimized along the normal direction based on a pre-determined objective function to update the initial facial mesh and obtain a backup facial mesh; A re-projection verification operation is performed on the backup facial mesh to determine the refined mesh that meets the preset refinement conditions.

[0005] As an optional implementation, in the first aspect of the invention, generating a composite texture confidence map based on the decomposition results of all the cameras and a pre-determined algorithm for constructing multiple influence data includes: The initial facial mesh is projected onto the texture images of all the cameras to obtain camera projection data for one or more key points. The key points include the mesh vertices of the initial facial mesh or the texture sampling points of the texture image. The camera projection data for each key point includes the visible camera corresponding to each key point and the projection position of each key point on the corresponding visible camera. Based on the camera projection data of each key point and the pre-determined multi-influence data construction algorithm, the multi-influence data of each key point is calculated; and based on the multi-influence data of all key points, a composite texture confidence map is generated.

[0006] As an optional implementation, in the first aspect of the present invention, the step of calculating multiple influence data for each key point based on camera projection data of each key point and a pre-determined multiple influence data construction algorithm includes: Based on the projection position of each key point on the corresponding visible camera, the high-frequency detail layer corresponding to the visible camera, and the predetermined cross-camera consistency term construction algorithm, the cross-camera consistency term construction operation is performed on the key point to obtain the cross-camera consistency term data of the key point. Based on the projection position of each key point on the corresponding visible camera, the residual layer and anomaly response layer corresponding to the visible camera, and the pre-determined anomaly penalty term construction algorithm, anomaly penalty term construction operation is performed on the key point to obtain the anomaly penalty term data of the key point. Based on the texture image of the visible camera corresponding to each key point, the base reflection layer of the visible camera, and the pre-determined imaging quality term construction algorithm, an imaging quality term construction operation is performed on the key point to obtain the imaging quality term data of the key point. Based on the projection position of each key point on the corresponding visible camera, the pre-determined reference position of the key point, and the pre-determined reprojection error term construction algorithm, the reprojection error term construction operation is performed on the key point to obtain the reprojection error term data of the key point. Based on the preset semantic region label of each key point and the pre-determined region prior mapping function algorithm, a region prior construction operation is performed on the key point to obtain the region prior data of the key point; The multiple impact data for each key point include cross-camera consistency data, anomaly penalty data, image quality data, reprojection error data, and region prior data.

[0007] As an optional implementation, in the first aspect of the invention, the step of optimizing the displacement of the vertices of the mesh to be updated along the normal direction based on a predetermined objective function to update the initial facial mesh and obtain a backup facial mesh includes: Obtain the position of the vertex in the mesh to be updated before the update; Based on the predetermined objective function and the displacement range constraint conditions corresponding to the objective function, the required normal displacement of the vertex to be updated is determined according to the vertex position before the update of the vertex to be updated. For the vertex position before the update of the vertex to be updated, the displacement is optimized along the normal direction according to the normal displacement required by the vertex to be updated, to obtain the vertex position after the update of the vertex to be updated. The spare face mesh includes the vertex position after the update of the vertex to be updated and the vertex positions before the update of the remaining vertices of all the vertices except the vertex to be updated.

[0008] As an optional implementation, in the first aspect of the present invention, the method further includes: Based on the weight control mechanism corresponding to the predetermined objective function, the region to which the vertex to be updated belongs is determined according to the vertex position before the update of the vertex to be updated. The vertex to be updated whose region is a preset abnormal region is masked to obtain the target vertex whose region is a preset updatable region. Furthermore, the step of optimizing the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function to update the initial facial mesh and obtain a backup facial mesh includes: Based on a predetermined objective function, the target mesh vertices are displacement optimized along the normal direction to update the initial face mesh and obtain a backup face mesh.

[0009] As an optional implementation, in a first aspect of the invention, performing a re-projection verification operation on the backup facial mesh to determine a refined mesh that meets preset refinement conditions includes: The alternative facial mesh is projected onto the texture images of all the cameras to evaluate the alternative facial mesh and obtain the evaluation result; Based on the evaluation results, determine whether the current conditions of the backup facial mesh meet the preset refinement conditions; When it is determined that the current conditions of the backup face mesh do not meet the refinement conditions, the operation of re-executing the pre-determined objective function to optimize the displacement of the vertices of the mesh to be updated along the normal direction to update the initial face mesh and obtain the backup face mesh is performed. When it is determined that the current conditions of the backup facial mesh meet the refinement conditions, the backup facial mesh is identified as a refinement mesh that meets the preset refinement conditions.

[0010] As an optional implementation, in the first aspect of the present invention, the evaluation result includes at least one of the following: the magnitude of the decrease in the objective function, the magnitude of the improvement in the cross-camera consistency term, the reprojection error, the change in vertex normal displacement, and the current iteration number; And, the step of determining whether the current conditions of the backup facial mesh meet the preset refinement conditions based on the evaluation results includes: Determine whether the decrease in the objective function is less than a preset decrease, denoted as the first condition; or, Determine whether the improvement in the cross-camera consistency term is less than a preset improvement, and record this as the second condition; or, Determine whether the reprojection error is less than a preset error, and record this as the third condition; or, Determine whether the change in the normal displacement of the vertex is less than a preset change, denoted as the fourth condition; or, Determine whether the current iteration count has reached the preset maximum iteration count, denoted as the fifth condition; Specifically, when at least one of the first condition, the second condition, the third condition, the fourth condition, and the fifth condition is satisfied, it is determined that the current condition of the backup facial mesh meets the preset refinement condition.

[0011] A second aspect of the present invention discloses a mesh refinement device based on texture reliability and partition normal displacement, the device comprising: The texture confidence decomposition module is used to perform confidence decomposition on the texture image of the scanned face acquired by each of the multiple cameras to obtain the decomposition result of that camera. The decomposition result includes at least one of the base reflection layer, high frequency detail layer, residual layer and anomaly response layer. The composite credibility generation module is used to generate a composite texture credibility map based on the decomposition results of all the cameras and a pre-determined multiple influence data. The face partitioning control module is used to partition the initial face mesh of the scanned face based on the composite texture confidence map and semantic region information to obtain a face partitioning result. The partitioning result includes at least one of a refinement area, a boundary-restricted area, a preservation area, and an anomaly shielding area. The normal displacement optimization module is used to optimize the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function, based on the facial partitioning results, so as to update the initial facial mesh and obtain a backup facial mesh. The re-projection verification module is used to perform a re-projection verification operation on the backup facial mesh to determine the refined mesh that meets the preset refinement conditions.

[0012] As an optional implementation, in the second aspect of the present invention, the composite credibility generation module generates the composite texture credibility map according to the decomposition results of all the cameras and a pre-determined multiple influence data construction algorithm, specifically in the following ways: The initial facial mesh is projected onto the texture images of all the cameras to obtain camera projection data for one or more key points. The key points include the mesh vertices of the initial facial mesh or the texture sampling points of the texture image. The camera projection data for each key point includes the visible camera corresponding to each key point and the projection position of each key point on the corresponding visible camera. Based on the camera projection data of each key point and the pre-determined multi-influence data construction algorithm, the multi-influence data of each key point is calculated; and based on the multi-influence data of all key points, a composite texture confidence map is generated.

[0013] As an optional implementation, in the second aspect of the present invention, the specific method by which the composite credibility generation module calculates multiple influence data for each key point based on camera projection data for each key point and a pre-determined multiple influence data construction algorithm includes: Based on the projection position of each key point on the corresponding visible camera, the high-frequency detail layer corresponding to the visible camera, and the predetermined cross-camera consistency term construction algorithm, the cross-camera consistency term construction operation is performed on the key point to obtain the cross-camera consistency term data of the key point. Based on the projection position of each key point on the corresponding visible camera, the residual layer and anomaly response layer corresponding to the visible camera, and the pre-determined anomaly penalty term construction algorithm, anomaly penalty term construction operation is performed on the key point to obtain the anomaly penalty term data of the key point. Based on the texture image of the visible camera corresponding to each key point, the base reflection layer of the visible camera, and the pre-determined imaging quality term construction algorithm, an imaging quality term construction operation is performed on the key point to obtain the imaging quality term data of the key point. Based on the projection position of each key point on the corresponding visible camera, the pre-determined reference position of the key point, and the pre-determined reprojection error term construction algorithm, the reprojection error term construction operation is performed on the key point to obtain the reprojection error term data of the key point. Based on the preset semantic region label of each key point and the pre-determined region prior mapping function algorithm, a region prior construction operation is performed on the key point to obtain the region prior data of the key point; The multiple impact data for each key point include cross-camera consistency data, anomaly penalty data, image quality data, reprojection error data, and region prior data.

[0014] As an optional implementation, in a second aspect of the invention, the normal displacement optimization module optimizes the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function to update the initial facial mesh and obtain a backup facial mesh. The specific method includes: Obtain the position of the vertex in the mesh to be updated before the update; Based on the predetermined objective function and the displacement range constraint conditions corresponding to the objective function, the required normal displacement of the vertex to be updated is determined according to the vertex position before the update of the vertex to be updated. For the vertex position before the update of the vertex to be updated, the displacement is optimized along the normal direction according to the normal displacement required by the vertex to be updated, to obtain the vertex position after the update of the vertex to be updated. The spare face mesh includes the vertex position after the update of the vertex to be updated and the vertex positions before the update of the remaining vertices of all the vertices except the vertex to be updated.

[0015] As an optional implementation, in a second aspect of the invention, the apparatus further includes: The abnormal region masking module is used to determine the region to which the vertex to be updated belongs based on the pre-determined weight control mechanism corresponding to the objective function and the vertex position before the update of the vertex to be updated. The module then masks the vertex to be updated whose region is a preset abnormal region, thereby obtaining the target vertex whose region is a preset updatable region. Furthermore, the normal displacement optimization module optimizes the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function to update the initial facial mesh and obtain the backup facial mesh. The specific methods include: Based on a predetermined objective function, the target mesh vertices are displacement optimized along the normal direction to update the initial face mesh and obtain a backup face mesh.

[0016] As an optional implementation, in a second aspect of the invention, the method by which the re-projection verification module performs a re-projection verification operation on the spare facial mesh to determine the specific method of refining the mesh that meets the preset refinement conditions includes: The alternative facial mesh is projected onto the texture images of all the cameras to evaluate the alternative facial mesh and obtain the evaluation result; Based on the evaluation results, determine whether the current conditions of the backup facial mesh meet the preset refinement conditions; When it is determined that the current conditions of the backup face mesh do not meet the refinement conditions, the operation of re-executing the pre-determined objective function to optimize the displacement of the vertices of the mesh to be updated along the normal direction to update the initial face mesh and obtain the backup face mesh is performed. When it is determined that the current conditions of the backup facial mesh meet the refinement conditions, the backup facial mesh is identified as a refinement mesh that meets the preset refinement conditions.

[0017] As an optional implementation, in a second aspect of the invention, the evaluation results include at least one of the following: the magnitude of the decrease in the objective function, the magnitude of the improvement in the cross-camera consistency term, the reprojection error, the change in vertex normal displacement, and the current iteration number; Furthermore, the specific methods by which the feedback verification module determines whether the current conditions of the backup facial mesh meet the preset refinement conditions based on the evaluation results include: Determine whether the decrease in the objective function is less than a preset decrease, denoted as the first condition; or, Determine whether the improvement in the cross-camera consistency term is less than a preset improvement, and record this as the second condition; or, Determine whether the reprojection error is less than a preset error, and record this as the third condition; or, Determine whether the change in the normal displacement of the vertex is less than a preset change, denoted as the fourth condition; or, Determine whether the current iteration count has reached the preset maximum iteration count, denoted as the fifth condition; Specifically, when at least one of the first condition, the second condition, the third condition, the fourth condition, and the fifth condition is satisfied, it is determined that the current condition of the backup facial mesh meets the preset refinement condition.

[0018] A third aspect of the present invention discloses another mesh refinement apparatus based on texture confidence and partition normal displacement, the apparatus comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the mesh refinement method based on texture confidence and partition normal displacement disclosed in the first aspect of the present invention.

[0019] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the mesh refinement method based on texture reliability and partition normal displacement disclosed in the first aspect of the present invention.

[0020] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment of the invention, the texture image of the scanned face acquired by each of the multiple cameras is subjected to confidence decomposition to obtain the decomposition result of that camera. The decomposition result includes at least one of a base reflection layer, a high-frequency detail layer, a residual layer, and an anomaly response layer. Based on the decomposition results of all the cameras and a pre-determined multiple influence data construction algorithm, a composite texture confidence map is generated. Based on the composite texture confidence map and semantic region information, the initial facial mesh of the scanned face is partitioned to obtain a facial partitioning result. The partitioning result includes at least one of a refinement region, a boundary-restricted region, a preservation region, and an anomaly shielding region. For the vertices of the mesh to be updated determined from the facial partitioning result, the vertices of the mesh to be updated are displacement-optimized along the normal direction based on a pre-determined objective function to update the initial facial mesh, obtaining a backup facial mesh. A back-projection verification operation is performed on the backup facial mesh to determine a refined mesh that meets the preset refinement conditions. As can be seen, implementing this invention can effectively separate illumination, noise, and real facial texture by decomposing texture images acquired by multiple cameras into a base reflection layer, a high-frequency detail layer, a residual layer, and an anomaly response layer, thereby improving the purity and usability of texture information from the source. Based on the multi-camera decomposition results and multiple influence data, a composite texture credibility map is generated, enabling the fusion and complementarity of multi-view information and avoiding information loss and bias under a single camera view. Furthermore, by combining the composite texture credibility map with semantic region information to divide the face into refinement areas, boundary-restricted areas, preservation areas, and anomaly-masking areas, differentiated processing strategies for different facial regions can be implemented, ensuring the safety and specificity of geometric editing. Subsequently, displacement optimization is performed on the vertices of the mesh to be updated along the normal direction based on the objective function. This approach can accurately recover high-frequency geometric details while maintaining the overall rationality of the facial structure. Finally, through a re-projection verification operation, refined meshes that meet the preset refinement conditions are selected, which can effectively eliminate abnormal geometry caused by erroneous optimization, thus ensuring the robustness and high fidelity of the final mesh. This solution constructs a facial mesh refinement process based on multi-camera texture credibility decomposition and partitioned restricted normal displacement. It can fully utilize the redundant information of multiple cameras to improve reconstruction quality, and can balance efficiency and accuracy through layered decomposition and partitioning strategies. Ultimately, it can generate high-quality facial meshes with accurate geometric structure, rich texture details, and no abnormal artifacts, which is conducive to improving the accuracy of facial mesh refinement and thus improving the scanning accuracy of refined facial meshes in scenarios such as 3D face reconstruction and digital human driving. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a mesh refinement method based on texture reliability and partitioned normal displacement disclosed in an embodiment of the present invention. Figure 2 This is a flowchart illustrating another mesh refinement method based on texture reliability and partition normal displacement disclosed in an embodiment of the present invention. Figure 3 This is a schematic diagram of a facial mesh refinement process disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a mesh refinement device based on texture reliability and partition normal displacement disclosed in an embodiment of the present invention; Figure 5 This is a schematic diagram of another mesh refinement device based on texture reliability and partition normal displacement disclosed in an embodiment of the present invention; Figure 6 This is a schematic diagram of another mesh refinement device based on texture reliability and partition normal displacement disclosed in an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] This invention discloses a mesh refinement method and apparatus based on texture credibility and partitioned normal displacement. It can decompose texture images acquired by multiple cameras into a base reflection layer, a high-frequency detail layer, a residual layer, and an anomaly response layer, effectively separating illumination, noise, and real facial texture, thus improving the purity and usability of texture information from the source. Based on the multi-camera decomposition results and multiple influence data construction algorithms, a composite texture credibility map is generated, enabling the fusion and complementarity of multi-view information and avoiding information loss and bias from a single camera perspective. Furthermore, by combining the composite texture credibility map with semantic region information, the face is partitioned into refinement areas, boundary-restricted areas, preservation areas, and anomaly-masking areas, enabling differentiated processing strategies for different facial regions, ensuring the safety and specificity of geometric editing. Subsequently, based on the target... The standard function performs displacement optimization along the normal direction, which can accurately recover high-frequency geometric details while maintaining the rationality of the overall facial structure. Finally, the refined mesh that meets the preset refinement conditions is screened through a re-projection verification operation, which can effectively eliminate abnormal geometry caused by erroneous optimization, and help ensure the robustness and high fidelity of the final mesh. This solution constructs a facial mesh refinement process with multi-camera texture confidence decomposition and partitioned restricted normal displacement. It can make full use of the redundant information of multiple cameras to improve the reconstruction quality, and can balance efficiency and accuracy through layered decomposition and partitioning strategies. In the end, it can generate a high-quality facial mesh with accurate geometric structure, rich texture details and no abnormal artifacts, which can improve the accuracy of facial mesh refinement, and thus improve the scanning accuracy of refined facial mesh in scenarios such as face 3D reconstruction and digital human driving. The following are detailed descriptions.

[0027] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating a mesh refinement method based on texture reliability and partitioned normal displacement disclosed in an embodiment of the present invention. Figure 1 The described mesh refinement method based on texture confidence and partitioned normal displacement can be applied to a mesh refinement device based on texture confidence and partitioned normal displacement. This device may include a refinement apparatus or a refinement server, wherein the refinement server may include a cloud server or a local server; this embodiment of the invention does not impose limitations. Figure 1As shown, this mesh refinement method based on texture confidence and partition normal displacement can include the following operations: 101. Perform confidence decomposition on the texture image of the scanned face acquired by each of the multiple cameras to obtain the decomposition result of that camera.

[0028] The mesh refinement method based on texture confidence and partition normal displacement described in this embodiment of the invention includes at least the following input parameters: An initial 3D facial mesh of the scanned face generated by a depth imaging device or a 3D reconstruction system; Facial texture images of the scanned face captured by multiple cameras; Calibration parameters and projection relationships of each camera; Facial semantic segmentation structure, key point localization results, anomaly detection results, etc.

[0029] The initial 3D facial mesh provides the basic geometry; texture images from multiple cameras provide evidence of multi-view appearance details; calibration parameters establish the mapping relationship between mesh vertices and image pixels; semantic segmentation, keypoints, and anomaly detection results assist in partitioning control and anomaly region identification. The projection relationship between the texture images from multiple cameras and the initial 3D facial mesh is as follows: Figure 2 As shown, Figure 2 This invention discloses a multi-camera acquisition and projection relationship diagram, wherein texture images T1, T2, and TN of the scanned face are acquired by cameras 1, 2, and N (N can be an integer greater than 2), respectively. The initial three-dimensional facial mesh (i.e., obtained through depth imaging or three-dimensional reconstruction) is then used to... Figure 2 The initial coarse mesh shown is projected onto the texture images of each camera. The mapping of each mesh vertex to the multi-camera image is achieved through texture domain sampling. The projection process combines the calibration parameters (including intrinsic and extrinsic parameters) of each camera and the projection relationship.

[0030] In this embodiment of the invention, optionally, the decomposition result includes at least one of a base reflection layer, a high-frequency detail layer, a residual layer, and an anomaly response layer. The residual layer can be an illumination residual layer or a specular residual layer; this embodiment of the invention does not limit the specific type of residual layer.

[0031] Specifically, let the texture image of the k-th camera be... The decomposition result is as follows:

[0032] in, It serves as a base reflective layer, used to characterize stable components such as low-frequency skin tone and diffuse reflection base color; This is a high-frequency detail layer, used to characterize detail changes that may be related to the real geometry; As an illumination / specular residual layer, it is used to characterize unstable components such as highlights, shadows, and specular reflections; This is an abnormal response layer used to characterize abnormal areas such as hair, occlusion, blur, overexposure, and localized contamination.

[0033] 102. Based on the decomposition results of all cameras and the pre-determined multiple influence data, construct an algorithm to generate a composite texture confidence map.

[0034] In this embodiment of the invention, optionally, the multiple influence data construction algorithms may include at least two of the following: cross-camera consistency term construction algorithm, anomaly penalty term construction algorithm, imaging quality term construction algorithm, reprojection error term construction algorithm, and region prior mapping function algorithm. This embodiment of the invention does not limit the specific algorithms.

[0035] 103. Based on the composite texture confidence map and semantic region information, perform facial partitioning on the initial facial mesh of the scanned face to obtain the facial partitioning results.

[0036] In this embodiment of the invention, the partitioning result includes at least one of a refinement region, a boundary-restricted region, a preservation region, and an anomaly shielding region, which can be specifically referred to as a high-confidence refinement region, a boundary-restricted region, a low-confidence preservation region, and an anomaly shielding region, respectively.

[0037] 104. For the vertices of the mesh to be updated determined from the facial segmentation results, based on the pre-determined objective function, the vertices of the mesh to be updated are displacement optimized along the normal direction to update the initial facial mesh and obtain the backup facial mesh.

[0038] In this embodiment of the invention, the vertices to be updated represent vertices that are allowed to be updated. Specifically, for vertices that are allowed to be updated, displacement optimization is performed only along the normal direction, and displacement constraints are set according to the region label to which the vertex belongs and the local curvature, thereby updating and obtaining the backup face mesh.

[0039] 105. Perform a re-projection verification operation on the backup facial mesh to determine the refined mesh that meets the preset refinement conditions.

[0040] In this embodiment of the invention, the updated mesh, i.e. the backup facial mesh, is reprojected onto each camera image to re-evaluate consistency and error. Once the stopping condition is met, the final refined mesh is output.

[0041] For example, taking a rule-driven implementation: an initial coarse mesh is generated using structured light, speckle projection, binocular / multi-view vision, ToF, laser scanning, or other 3D reconstruction methods; facial texture images are simultaneously acquired using multiple cameras; high- and low-frequency decomposition results are obtained using Retinex or bilateral filtering; highlights, shadows, and blurred regions are detected based on brightness abrupt changes, local saturation, local contrast, and edge continuity; the consistency of high-frequency response of each camera to the same surface point is calculated; a composite texture confidence map is generated; partitioning is completed by combining semantic region information; normal displacement optimization is performed on high-confidence refinement areas and boundary-constrained areas; abnormal regions are frozen or removed; multiple iterations are performed through back-projection verification to output the refinement results.

[0042] For example, taking the fusion of existing semantic, keypoint, or anomaly modules as an example: the existing semantic segmentation module is called to obtain region labels such as skin area, periorbital area, nasal wing, and lip area; the existing keypoint localization module is called to assist in defining the lip edge, nasal wing edge, and periorbital boundary; the existing anomaly detection module is called to identify hair, occlusion, highlights, and blurred areas; the above priors are fused with the texture confidence map to form the final partition; normal displacement optimization is performed on the regions that are allowed to be updated; the data item contribution of anomaly regions is masked during the optimization stage; and the final refined mesh is output.

[0043] It is evident that implementation Figure 1The described mesh refinement method based on texture credibility and partitioned normal displacement can effectively separate illumination, noise, and real facial texture by decomposing texture images acquired by multiple cameras into a basic reflection layer, a high-frequency detail layer, a residual layer, and anomaly response layer, thereby improving the purity and usability of texture information from the source. Based on the multi-camera decomposition results and multiple influence data, a composite texture credibility map is generated, enabling the fusion and complementarity of multi-view information and avoiding information loss and bias from a single-camera perspective. Furthermore, by combining the composite texture credibility map with semantic region information, the face is partitioned into refinement areas, boundary-restricted areas, preservation areas, and anomaly-masking areas, enabling differentiated processing strategies for different facial regions, ensuring the safety and specificity of geometric editing. Subsequently, for the vertices of the mesh to be updated, the objective function is applied along... Normal displacement optimization can accurately restore high-frequency geometric details while maintaining the rationality of the overall facial structure. Finally, the refined mesh that meets the preset refinement conditions is screened through the re-projection verification operation, which can effectively eliminate abnormal geometry caused by erroneous optimization, and help ensure the robustness and high fidelity of the final mesh. This solution constructs a facial mesh refinement process with multi-camera texture credibility decomposition and partitioned restricted normal displacement. It can make full use of the redundant information of multi-camera to improve the reconstruction quality, and can balance efficiency and accuracy through layer decomposition and partitioning strategies. In the end, it can generate a high-quality facial mesh with accurate geometric structure, rich texture details and no abnormal artifacts, which can improve the accuracy of facial mesh refinement, and thus improve the scanning accuracy of refined facial mesh in scenarios such as face 3D reconstruction and digital human driving.

[0044] In an optional embodiment, step 102 above, which generates a composite texture confidence map based on the decomposition results of all cameras and a pre-determined algorithm for constructing multiple influence data, includes: The initial facial mesh is projected onto the texture images of all cameras to obtain camera projection data for one or more key points. The key points include the mesh vertices of the initial facial mesh or the texture sampling points of the texture image. The camera projection data of each key point includes the visible camera corresponding to each key point and the projection position of each key point on the corresponding visible camera. Based on the camera projection data of each key point and the pre-determined multiple influence data, an algorithm is constructed to calculate the multiple influence data of each key point; and based on the multiple influence data of all key points, a composite texture confidence map is generated.

[0045] In this embodiment of the invention, optionally, the multiple influencing data include at least two of the following: cross-camera consistency data, anomaly penalty data, image quality data, reprojection error data, and region prior data. This embodiment of the invention does not limit the types of data.

[0046] As can be seen, this optional embodiment can obtain camera projection data of key points by projecting the initial facial mesh onto all camera texture images, establish a precise correspondence between geometry and multi-view textures, and clarify the visibility and projection position of each key point. Based on the projection data of each key point, it combines multiple influence data to construct an algorithm to calculate its multiple influence data and generate a composite texture confidence map. It can fully integrate multi-dimensional influence factors such as visibility and consistency under multiple camera views, so that the confidence map can accurately reflect the true reliability of the texture in each region. This is beneficial for providing an accurate data foundation for subsequent facial partitioning based on semantic region information and accurate selection and normal displacement optimization of the vertices of the mesh to be updated. This helps to ensure that the differentiated refinement strategy can be accurately applied to the reliable region and avoid the abnormal region.

[0047] In this optional embodiment, as an optional implementation, based on the camera projection data of each key point and a pre-determined algorithm for constructing multiple influence data, multiple influence data of each key point are calculated, including: Based on the projection position of each keypoint on the corresponding visible camera, the high-frequency detail layer corresponding to the visible camera, and the pre-determined cross-camera consistency term construction algorithm, the cross-camera consistency term construction operation is performed on the keypoint to obtain the cross-camera consistency term data of the keypoint. Based on the projection position of each key point on the corresponding visible camera, the residual layer and anomaly response layer corresponding to the visible camera, and the pre-determined anomaly penalty term construction algorithm, the anomaly penalty term construction operation is performed on the key point to obtain the anomaly penalty term data of the key point. Based on the texture image of the visible camera corresponding to each key point, the basic reflection layer of the visible camera, and the pre-determined imaging quality term construction algorithm, the imaging quality term construction operation is performed on the key point to obtain the imaging quality term data of the key point. Based on the projection position of each key point on the corresponding visible camera, the pre-determined reference position of the key point, and the pre-determined reprojection error term construction algorithm, the reprojection error term construction operation is performed on the key point to obtain the reprojection error term data of the key point. Based on the preset semantic region label of each key point and the pre-determined region prior mapping function algorithm, the region prior construction operation is performed on the key point to obtain the region prior data of the key point; Among them, the multiple impact data for each key point include cross-camera consistency data, anomaly penalty data, image quality data, reprojection error data, and region prior data.

[0048] In this embodiment of the invention, specifically, based on the projection position of each keypoint on the camera, the detail response amplitude and gradient direction are extracted from the high-frequency detail layer. Then, based on the detail response amplitude and gradient direction, and a pre-determined cross-camera consistency term construction algorithm, a cross-camera consistency term construction operation is performed on the keypoint to obtain the cross-camera consistency term data for that keypoint. The cross-camera consistency term data is used to measure whether the high-frequency detail response of the same surface point is consistent under different viewpoints.

[0049] The formula for calculating the detail response amplitude is as follows:

[0050] in, Key point In the camera The projection position on the surface Key point The detailed response amplitude.

[0051] The formula for calculating the gradient direction is as follows:

[0052] in, Key point gradient direction, To prevent extremely small constants with a denominator of zero.

[0053] The cross-camera consistency term construction algorithm includes:

[0054] in, Key point Cross-camera consistency data, Key point The set of camera pairs in the visible cameras This is the normalization parameter for amplitude difference.

[0055] In this embodiment of the invention, the algorithm for constructing the anomaly penalty term includes:

[0056] in, Key point Abnormal penalty data, Key point The set of visible cameras, , For normalized mapping functions, , This represents the weighting coefficient. The larger the anomaly penalty data, the higher the likelihood that the keypoint is affected by interference such as highlights, shadows, hair, occlusion, blur, and overexposure, and the lower its geometric update reliability should be.

[0057] In this embodiment of the invention, the image quality term construction algorithm can be constructed based on indicators such as sharpness, local contrast, and exposure stability, as follows:

[0058] in, For image quality data, For local sharpness indicators, This is a local contrast index. To expose quality indicators, All are weights.

[0059] In this embodiment of the invention, the reprojection error term is used to measure the degree of geometric matching between the current mesh and the multi-camera texture, and the reprojection error term construction algorithm includes:

[0060] in, Key point The reprojection error term data, Key point In the Projection position in each camera Key point In the The corresponding reference pixel position or matching position in each camera.

[0061] Furthermore, the multi-camera comprehensive reprojection error term can be determined by combining the reprojection error term data of key points from multiple cameras. The larger this term is, the more stable the projection matching between the current geometry and the multi-view texture is. The algorithm for this multi-camera comprehensive reprojection error term includes:

[0062] in, Key point Multi-camera composite reprojection error. These are the normalized parameters.

[0063] In this embodiment of the invention, region prior data is used to control the degree of conservatism in updating different regions based on facial anatomy and semantic partitioning, and the region prior construction algorithm includes:

[0064] in, Key point Regional prior data, Key point Semantic region tag, This is the region prior mapping function.

[0065] For example, continuous and stable areas such as the forehead, cheeks, and bridge of the nose can be assigned higher prior values; sensitive areas such as the edge of the nose, lip edge, periorbital contour, and jawline can be assigned lower prior values; and abnormal areas such as hair-covered areas, eyelash areas, and areas with obvious reflections can be assigned extremely low prior values ​​or be directly masked.

[0066] In this embodiment of the invention, the algorithm for constructing the composite texture confidence level corresponding to the composite texture confidence level graph includes:

[0067] in, Key point The credibility of composite textures For each weight, This can be a sigmoid function or other normalization function. The weights for each term can be empirical weights assigned during the initial engineering implementation based on the stability requirements of the facial scanning scenario. For example: assigning a higher weight to the cross-camera consistency term; assigning a stronger suppression weight to the anomaly penalty term; assigning a moderate weight to the semantic prior term; and assigning a stability correction weight to the reprojection error term.

[0068] Optionally, evaluation criteria such as reprojection error, boundary drift, local curvature recovery error, and patch folding rate can be used on the validation set to perform grid search, Bayesian optimization, or manual iterative parameter tuning on the weights of each item. For example, the following parameter optimization problem can be solved:

[0069] in, To verify the sample size, Let be the reprojection error of the nth sample. For boundary drift error, For normal recovery error, Penalty for face folding or topological anomalies, The weights for the evaluation items.

[0070] Optionally, the weights can also be set as learnable parameters, which can be jointly optimized through the training process to adaptively match the characteristics of different devices, different lighting conditions and different face regions.

[0071] As can be seen, this optional implementation can accurately calculate multiple influence data based on the projection position of each key point on the corresponding visible camera, the decomposition results of each camera, and the corresponding multiple influence data construction algorithm, thereby improving the accuracy of the subsequent generation of composite confidence maps based on multiple influence data.

[0072] In another optional embodiment, step 104 above, based on a pre-determined objective function, optimizes the displacement of the vertices of the mesh to be updated along the normal direction to update the initial face mesh and obtain a backup face mesh, including: Get the position of the vertex in the mesh to be updated before the update; Based on the predetermined objective function and the corresponding displacement range constraints, the required normal displacement of the vertex to be updated is determined according to the vertex position before the update of the vertex to be updated. Before updating the vertex position of the mesh to be updated, perform displacement optimization along the normal direction according to the normal displacement required by the vertex of the mesh to be updated, to obtain the updated vertex position of the mesh to be updated. The backup face mesh includes the updated vertex position of the mesh to be updated and the original vertex positions of the remaining mesh vertices excluding the mesh to be updated.

[0073] In this embodiment of the invention, the normal displacement is obtained through joint optimization of texture-driven terms, region constraint terms, smoothing terms, and boundary terms, and the calculation formula for the normal displacement is as follows:

[0074] in, For vertices that are allowed to be updated, For the corresponding position before the update, For the corresponding normal direction, This is the corresponding updated position.

[0075] Specifically, for the normal displacement, the following objective function can be constructed. :

[0076] in, For texture-driven data items, For smoothing terms, For boundary preservation terms, It is a weak symmetric constraint term. For topological stability terms, , , , The weights of each regularization term.

[0077] The calculation formula for the texture-driven data item is as follows:

[0078] in, As vertices Texture-driven data item weights, This represents the geometric consistency error of vertices under multi-camera texture constraints.

[0079] Preferably, the weights of texture-driven data items can be written as:

[0080] in, Labels for vertex regions. For regional coefficients, The credibility of the vertex composite texture.

[0081] The formula for calculating the smoothing term is as follows:

[0082] in, It is a set of adjacent vertex pairs.

[0083] Boundary preservation terms can be used to suppress excessive updates in areas such as the lip margin, nasal alar edge, and periorbital contour. The formula for calculating boundary preservation terms is as follows:

[0084] in, For the set of vertices in the boundary-sensitive region, This is the boundary penalty coefficient.

[0085] In this embodiment of the invention, the displacement range constraint condition may include:

[0086] in, Labels for vertex regions. This refers to local curvature, normal variation, or boundary proximity. For example, the high-confidence refinement region allows for relatively large continuous micro-displacements; the boundary-constrained region has a smaller displacement limit; the low-confidence maintenance region only allows for minimal displacement; and the anomaly shielding region can directly set... Normal displacement The objective function and constraints described above are used to solve the problem.

[0087] As can be seen, this optional embodiment can determine the required normal displacement of the vertex to be updated based on the vertex position before the update of the vertex to be updated by constructing an objective function and the corresponding displacement range constraint condition. Then, it can optimize the displacement of the vertex based on the normal displacement to obtain the updated vertex position, thereby updating the initial three-dimensional face mesh. This can improve the accuracy and reliability of the determination of the normal displacement, thereby improving the accuracy and reliability of the displacement optimization of the vertex to be updated based on the normal displacement, and further improving the accuracy and reliability of the updated backup face mesh.

[0088] Example 2 Please see Figure 2 , Figure 2 This is a flowchart illustrating a mesh refinement method based on texture reliability and partitioned normal displacement disclosed in an embodiment of the present invention. Figure 2 The described mesh refinement method based on texture confidence and partitioned normal displacement can be applied to a mesh refinement device based on texture confidence and partitioned normal displacement. This device may include a refinement apparatus or a refinement server, wherein the refinement server may include a cloud server or a local server; this embodiment of the invention does not impose limitations. Figure 2 As shown, this mesh refinement method based on texture confidence and partition normal displacement can include the following operations: 201. Perform confidence decomposition on the texture image of the scanned face acquired by each of the multiple cameras to obtain the decomposition result of that camera.

[0089] 202. Based on the decomposition results of all cameras and the pre-determined multiple influence data, construct an algorithm to generate a composite texture confidence map.

[0090] 203. Based on the composite texture confidence map and semantic region information, perform facial partitioning on the initial facial mesh of the scanned face to obtain the facial partitioning results.

[0091] 204. For the vertices of the mesh to be updated determined from the facial partitioning results, based on the weight control mechanism corresponding to the pre-determined objective function, the region to which the vertices of the mesh to be updated belong is determined according to the vertex position of the vertices of the mesh to be updated before the update, so as to mask the vertices of the mesh to be updated whose region is a preset abnormal region, and obtain the target mesh vertices whose region is a preset updatable region.

[0092] In this embodiment of the invention, a weight control mechanism is implemented during the objective function construction stage to shield, freeze, or reduce the weight of abnormal regions during geometric optimization. Specifically, assuming vertices... The exception label is denoted as ,in: , indicating the normal area, Indicates a region of mild abnormality. This indicates a severely abnormal area.

[0093] The weight of the corresponding data item can then be defined as:

[0094] in, This represents the weighting factor. Correspondingly, the weights of texture-driven data items can be written as:

[0095] In other words, for severely anomalous regions, their data item weights are directly set to zero, preventing them from participating in texture-driven updates; for mildly anomalous regions, a weight reduction approach is used for weak participation in optimization; and for normal regions, they participate in optimization normally based on region coefficients and confidence levels. Meanwhile, anomalous regions can still participate in smoothing or topological stability terms to maintain local geometric continuity. This allows the texture confidence decomposition results to be fed into the cross-camera consistency term, anomaly penalty term, imaging quality term, and composite confidence graph construction process, respectively. The composite confidence level further determines whether vertices are allowed to be updated and their data item weights. The normal displacement is solved under the combined effect of the objective function and its partitioning constraints, smoothing constraints, and boundary constraints. Anomalous regions are shielded during the objective function modeling stage through freezing or weight reduction mechanisms, thus forming a complete closed-loop technical chain encompassing texture evidence screening, confidence fusion, partitioning control, restricted displacement optimization, and regression verification convergence.

[0096] 205. Based on the predetermined objective function, the target mesh vertices are displacement optimized along the normal direction to update the initial face mesh and obtain the backup face mesh.

[0097] 206. Perform a re-projection verification operation on the backup facial mesh to determine the refined mesh that meets the preset refinement conditions.

[0098] In this embodiment of the invention, for other descriptions of steps 201-203 and steps 205-206, please refer to the detailed description of steps 101-105 in Embodiment 1. This embodiment of the invention is not limited.

[0099] It is evident that implementation Figure 2The described mesh refinement method based on texture credibility and partitioned normal displacement can effectively separate illumination, noise, and real facial texture by decomposing texture images acquired by multiple cameras into a basic reflection layer, a high-frequency detail layer, a residual layer, and anomaly response layer, thereby improving the purity and usability of texture information from the source. Based on the multi-camera decomposition results and multiple influence data, a composite texture credibility map is generated, enabling the fusion and complementarity of multi-view information and avoiding information loss and bias from a single-camera perspective. Furthermore, by combining the composite texture credibility map with semantic region information, the face is partitioned into refinement areas, boundary-restricted areas, preservation areas, and anomaly-masking areas, enabling differentiated processing strategies for different facial regions, ensuring the safety and specificity of geometric editing. Subsequently, for the vertices of the mesh to be updated, the objective function is applied along... Normal displacement optimization can accurately restore high-frequency geometric details while maintaining the rationality of the overall facial structure. Finally, the refined mesh that meets the preset refinement conditions is screened through the re-projection verification operation, which can effectively eliminate abnormal geometry caused by erroneous optimization, and help ensure the robustness and high fidelity of the final mesh. This solution constructs a facial mesh refinement process with multi-camera texture credibility decomposition and partitioned restricted normal displacement. It can make full use of the redundant information of multi-camera to improve the reconstruction quality, and can balance efficiency and accuracy through layer decomposition and partitioning strategies. In the end, it can generate a high-quality facial mesh with accurate geometric structure, rich texture details and no abnormal artifacts, which can improve the accuracy of facial mesh refinement, and thus improve the scanning accuracy of refined facial mesh in scenarios such as face 3D reconstruction and digital human driving. Furthermore, based on the weight control mechanism corresponding to the objective function, the region to which the vertex belongs can be determined as an abnormal region or an updatable region according to the vertex position of the vertex to be updated before the update. Thus, the vertex to be updated in the abnormal region can be masked. The combination of face partitioning and the weight control mechanism corresponding to the objective function can perform double masking of abnormal regions, which helps to improve the accuracy and reliability of the vertex to be updated determined after masking. It also reduces the occurrence of unnecessary erroneous optimization of vertices in abnormal regions, which leads to a decrease in mesh refinement efficiency. This is conducive to further improving the accuracy, reliability and efficiency of mesh refinement.

[0100] In an optional embodiment, step 206 above, which involves performing a re-projection verification operation on the backup facial mesh to determine the refined mesh that meets the preset refinement conditions, includes: The alternative face mesh is projected onto the texture images of all cameras to evaluate the alternative face mesh and obtain the evaluation results; Based on the evaluation results, determine whether the current conditions of the backup facial mesh meet the preset refinement conditions; When it is determined that the current conditions of the backup face mesh do not meet the refinement conditions, the operation of re-executing the operation based on the pre-determined objective function to optimize the displacement of the vertices of the mesh to be updated along the normal direction is performed to update the initial face mesh and obtain the backup face mesh. When it is determined that the current conditions of the backup face mesh meet the refinement conditions, the backup face mesh is identified as the refinement mesh that meets the preset refinement conditions.

[0101] In this embodiment of the invention, the evaluation results include at least one of the following: the magnitude of the decrease in the objective function, the magnitude of the improvement in the cross-camera consistency term, the reprojection error, the change in vertex normal displacement, and the current iteration number.

[0102] As can be seen, this optional embodiment can evaluate the mesh by projecting the backup facial mesh onto the texture images of all cameras, comprehensively measuring the mesh's fit to the real texture from multiple perspectives, avoiding evaluation bias caused by a single perspective, and thus obtaining objective and reliable evaluation results. Furthermore, based on the evaluation results, it can determine whether the refinement conditions are met, enabling automated quantitative screening of mesh optimization quality to ensure that a mesh that achieves the expected accuracy is obtained. When the refinement conditions are not met, displacement optimization is performed again along the normal direction based on the objective function, and a new backup mesh is iteratively generated. This closed-loop feedback mechanism allows the mesh to gradually approach the optimal solution, which is beneficial for continuously improving the geometric accuracy and texture consistency of the mesh. When the refinement conditions are met, the backup mesh is determined as the refined mesh, which is beneficial for providing a high-quality, high-fidelity facial mesh foundation for subsequent applications, such as ensuring that subsequent rendering, animation, or identity recognition operations based on this refined mesh have higher accuracy and realism.

[0103] In this optional embodiment, as an optional implementation, determining whether the current conditions of the backup facial mesh meet the preset refinement conditions based on the evaluation results includes: Determine whether the decrease in the objective function is less than a preset decrease, denoted as the first condition; or, Determine whether the improvement in cross-camera consistency is less than a preset improvement, and record this as the second condition; or... Determine whether the reprojection error is less than the preset error, denoted as the third condition; or... Determine whether the change in the normal displacement of the vertex is less than a preset change, denoted as the fourth condition; or... Determine whether the current iteration count has reached the preset maximum iteration count, denoted as the fifth condition; Specifically, when at least one of the first, second, third, fourth, and fifth conditions is met, it is determined that the current condition of the backup facial mesh meets the preset refinement condition.

[0104] In this embodiment of the invention, the refinement conditions can be determined using a single condition or a combination of multiple conditions to balance optimization convergence, result stability, and engineering computational cost. Optionally, the refinement conditions may include at least one of the following five conditions: 1. The objective function decreases less than the preset decrease range in two or more consecutive iterations; 2. The improvement in multi-camera consistency is lower than the preset improvement. 3. The reprojection error has decreased to below the preset error; 4. The average or maximum normal displacement of the vertex is lower than the preset change. 5. Reach the preset maximum number of iterations.

[0105] For example, suppose the overall objective function of the t-th iteration is:

[0106] The objective function's convergence stopping condition (i.e., the first condition mentioned above) corresponding to the magnitude of the objective function's decrease can be denoted as:

[0107] in, The threshold for the relative change of the objective function is used to represent the preset decrease range; To prevent extremely small constants with a denominator of zero.

[0108] In this embodiment of the invention, if the vertex normal displacement is denoted as Then the displacement convergence condition corresponding to the change in normal displacement at the vertex (i.e., the fourth condition mentioned above) can be written as:

[0109] or:

[0110] in, as well as All of these represent preset changes.

[0111] If the average reprojection error is denoted as , then the error stopping condition corresponding to the reprojection error (i.e., the third condition mentioned above) can be set as follows:

[0112] Additionally, a maximum iteration count constraint can be set, specifically: Stop iterating when the time comes.

[0113] For example, a convergence stopping condition can be set when the relative decrease of the objective function over two consecutive iterations is less than 1%. Alternatively, a refinement condition can be met when the average reprojection error is less than 0.3 pixels or the average normal displacement change at the vertex is less than 0.02 mm. It should be noted that these thresholds can be pre-calibrated or empirically set based on device accuracy, camera resolution, and the cleanliness of the operating channel.

[0114] As can be seen, this optional implementation can determine whether the mesh meets the refinement conditions based on at least one evaluation index, such as the decrease in the objective function of the updated mesh, the improvement in the cross-camera consistency term, the reprojection error, the change in vertex normal displacement, and the current iteration number. It can improve the flexibility, diversity, and accuracy of the judgment on whether the mesh meets the refinement conditions based on diverse evaluation indexes, reduce the occurrence of delays in determining whether the mesh has not been refined due to errors in the evaluation of a single index, and help improve the accuracy, efficiency, and reliability of mesh refinement.

[0115] For example, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a facial mesh refinement process disclosed in an embodiment of the present invention. The facial mesh refinement process includes: taking an initial coarse mesh and multi-camera texture as input, and performing processes such as reliability decomposition, partition control, normal priority, and anomaly masking through the control link of the present invention, thereby realizing the restoration of details such as the nose wing, lip edge, nasolabial folds, and eye area, and reducing drift, bulge, folding, and self-intersection to ensure boundary and topological stability, and then outputting a refined three-dimensional facial mesh.

[0116] Example 3 Please see Figure 4 , Figure 4 This is a schematic diagram of a mesh refinement device based on texture reliability and partition normal displacement disclosed in an embodiment of the present invention. Figure 4 The described mesh refinement apparatus based on texture reliability and partitioned normal displacement may include a refinement device or a refinement server, wherein the refinement server may include a cloud server or a local server, and the embodiments of the present invention are not limited thereto. Figure 4 As shown, the mesh refinement device based on texture confidence and partition normal displacement may include: The texture confidence decomposition module 301 is used to perform confidence decomposition on the texture image of the scanned face acquired by each of the multiple cameras to obtain the decomposition result of that camera. The decomposition result includes at least one of the base reflection layer, high frequency detail layer, residual layer and abnormal response layer.

[0117] The composite credibility generation module 302 is used to generate a composite texture credibility map based on the decomposition results of all cameras and the pre-determined multiple influence data.

[0118] The face partitioning control module 303 is used to partition the initial face mesh of the scanned face based on the composite texture confidence map and semantic region information to obtain face partitioning results. The partitioning results include at least one of the following: refinement area, boundary restricted area, preservation area and abnormal shielding area.

[0119] The normal displacement optimization module 304 is used to optimize the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function, based on the results of facial partitioning, so as to update the initial facial mesh and obtain a backup facial mesh.

[0120] The re-projection verification module 305 is used to perform a re-projection verification operation on the backup facial mesh to determine the refined mesh that meets the preset refinement conditions.

[0121] It is evident that implementation Figure 4 The described mesh refinement device based on texture credibility and partitioned normal displacement can effectively separate illumination, noise, and real facial texture by decomposing texture images acquired by multiple cameras into a base reflection layer, a high-frequency detail layer, a residual layer, and an anomaly response layer, thereby improving the purity and usability of texture information from the source. Based on the multi-camera decomposition results and multiple influence data, a composite texture credibility map is generated, enabling the fusion and complementarity of multi-view information and avoiding information loss and bias from a single-camera perspective. Furthermore, by combining the composite texture credibility map with semantic region information, the face is partitioned into refinement areas, boundary-restricted areas, preservation areas, and anomaly-masking areas, enabling differentiated processing strategies for different facial regions, ensuring the safety and specificity of geometric editing. Subsequently, based on the objective function along the vertices of the mesh to be updated... Normal displacement optimization can accurately restore high-frequency geometric details while maintaining the rationality of the overall facial structure. Finally, the refined mesh that meets the preset refinement conditions is screened through the re-projection verification operation, which can effectively eliminate abnormal geometry caused by erroneous optimization, and help ensure the robustness and high fidelity of the final mesh. This solution constructs a facial mesh refinement process with multi-camera texture credibility decomposition and partitioned restricted normal displacement. It can make full use of the redundant information of multi-camera to improve the reconstruction quality, and can balance efficiency and accuracy through layer decomposition and partitioning strategies. In the end, it can generate a high-quality facial mesh with accurate geometric structure, rich texture details and no abnormal artifacts, which can improve the accuracy of facial mesh refinement, and thus improve the scanning accuracy of refined facial mesh in scenarios such as face 3D reconstruction and digital human driving.

[0122] In an optional embodiment, the composite confidence generation module 302 generates the composite texture confidence map by constructing an algorithm based on the decomposition results of all cameras and pre-determined multiple influence data, specifically in the following ways: The initial facial mesh is projected onto the texture images of all cameras to obtain camera projection data for one or more key points. The key points include the mesh vertices of the initial facial mesh or the texture sampling points of the texture image. The camera projection data of each key point includes the visible camera corresponding to each key point and the projection position of each key point on the corresponding visible camera. Based on the camera projection data of each key point and the pre-determined multiple influence data, an algorithm is constructed to calculate the multiple influence data of each key point; and based on the multiple influence data of all key points, a composite texture confidence map is generated.

[0123] As can be seen, this optional embodiment can obtain camera projection data of key points by projecting the initial facial mesh onto all camera texture images, establish a precise correspondence between geometry and multi-view textures, and clarify the visibility and projection position of each key point. Based on the projection data of each key point, it combines multiple influence data to construct an algorithm to calculate its multiple influence data and generate a composite texture confidence map. It can fully integrate multi-dimensional influence factors such as visibility and consistency under multiple camera views, so that the confidence map can accurately reflect the true reliability of the texture in each region. This is beneficial for providing an accurate data foundation for subsequent facial partitioning based on semantic region information and accurate selection and normal displacement optimization of the vertices of the mesh to be updated. This helps to ensure that the differentiated refinement strategy can be accurately applied to the reliable region and avoid the abnormal region.

[0124] In this optional embodiment, as an optional implementation, the composite credibility generation module 302 constructs an algorithm based on the camera projection data of each key point and the pre-determined multiple influence data to calculate the multiple influence data of each key point in the following specific ways: Based on the projection position of each keypoint on the corresponding visible camera, the high-frequency detail layer corresponding to the visible camera, and the pre-determined cross-camera consistency term construction algorithm, the cross-camera consistency term construction operation is performed on the keypoint to obtain the cross-camera consistency term data of the keypoint. Based on the projection position of each key point on the corresponding visible camera, the residual layer and anomaly response layer corresponding to the visible camera, and the pre-determined anomaly penalty term construction algorithm, the anomaly penalty term construction operation is performed on the key point to obtain the anomaly penalty term data of the key point. Based on the texture image of the visible camera corresponding to each key point, the basic reflection layer of the visible camera, and the pre-determined imaging quality term construction algorithm, the imaging quality term construction operation is performed on the key point to obtain the imaging quality term data of the key point. Based on the projection position of each key point on the corresponding visible camera, the pre-determined reference position of the key point, and the pre-determined reprojection error term construction algorithm, the reprojection error term construction operation is performed on the key point to obtain the reprojection error term data of the key point. Based on the preset semantic region label of each key point and the pre-determined region prior mapping function algorithm, the region prior construction operation is performed on the key point to obtain the region prior data of the key point; Among them, the multiple impact data for each key point include cross-camera consistency data, anomaly penalty data, image quality data, reprojection error data, and region prior data.

[0125] As can be seen, this optional implementation can accurately calculate multiple influence data based on the projection position of each key point on the corresponding visible camera, the decomposition results of each camera, and the corresponding multiple influence data construction algorithm, thereby improving the accuracy of the subsequent generation of composite confidence maps based on multiple influence data.

[0126] In another optional embodiment, the normal displacement optimization module 304 optimizes the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function to update the initial face mesh and obtain the alternative face mesh. The specific methods include: Get the position of the vertex in the mesh to be updated before the update; Based on the predetermined objective function and the corresponding displacement range constraints, the required normal displacement of the vertex to be updated is determined according to the vertex position before the update of the vertex to be updated. Before updating the vertex position of the mesh to be updated, perform displacement optimization along the normal direction according to the normal displacement required by the vertex of the mesh to be updated, to obtain the updated vertex position of the mesh to be updated. The backup face mesh includes the updated vertex position of the mesh to be updated and the original vertex positions of the remaining mesh vertices excluding the mesh to be updated.

[0127] As can be seen, this optional embodiment can determine the required normal displacement of the vertex to be updated based on the vertex position before the update of the vertex to be updated by constructing an objective function and the corresponding displacement range constraint condition. Then, it can optimize the displacement of the vertex based on the normal displacement to obtain the updated vertex position, thereby updating the initial three-dimensional face mesh. This can improve the accuracy and reliability of the determination of the normal displacement, thereby improving the accuracy and reliability of the displacement optimization of the vertex to be updated based on the normal displacement, and further improving the accuracy and reliability of the updated backup face mesh.

[0128] In yet another alternative embodiment, such as Figure 5 As shown, Figure 5 This is a schematic diagram of another mesh refinement device based on texture reliability and partition normal displacement disclosed in an embodiment of the present invention, wherein the device may further include: The abnormal region shielding module 306 is used to determine the region to which the vertex to be updated belongs based on the pre-determined weight control mechanism corresponding to the objective function and the vertex position before the update of the vertex to be updated. The vertex to be updated whose region is a preset abnormal region is shielded to obtain the target vertex whose region is a preset updatable region. Furthermore, the normal displacement optimization module 304 optimizes the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function to update the initial face mesh and obtain the backup face mesh. The specific methods include: Based on a predetermined objective function, the target mesh vertices are displacement optimized along the normal direction to update the initial face mesh and obtain a backup face mesh.

[0129] It is evident that implementation Figure 5 The described device can also determine whether a region belongs to an abnormal region or an updatable region based on the weight control mechanism corresponding to the objective function and the position of the vertex to be updated before the update. This allows for the masking of the vertices to be updated in the abnormal region. The device can perform dual masking of abnormal regions by combining face partitioning and the weight control mechanism corresponding to the objective function. This helps to improve the accuracy and reliability of the vertices to be updated determined after masking, and reduces the occurrence of unnecessary erroneous optimization of vertices in abnormal regions that would reduce the efficiency of mesh refinement. This further improves the accuracy, reliability and efficiency of mesh refinement.

[0130] In yet another optional embodiment, the re-projection verification module 305 performs a re-projection verification operation on the backup facial mesh to determine the specific method by which the refined mesh that meets the preset refinement conditions is: The alternative face mesh is projected onto the texture images of all cameras to evaluate the alternative face mesh and obtain the evaluation results; Based on the evaluation results, determine whether the current conditions of the backup facial mesh meet the preset refinement conditions; When it is determined that the current conditions of the backup face mesh do not meet the refinement conditions, the operation of re-executing the operation based on the pre-determined objective function to optimize the displacement of the vertices of the mesh to be updated along the normal direction is performed to update the initial face mesh and obtain the backup face mesh. When it is determined that the current conditions of the backup face mesh meet the refinement conditions, the backup face mesh is identified as the refinement mesh that meets the preset refinement conditions.

[0131] As can be seen, this optional embodiment can evaluate the mesh by projecting the backup facial mesh onto the texture images of all cameras, comprehensively measuring the mesh's fit to the real texture from multiple perspectives, avoiding evaluation bias caused by a single perspective, and thus obtaining objective and reliable evaluation results. Furthermore, based on the evaluation results, it can determine whether the refinement conditions are met, enabling automated quantitative screening of mesh optimization quality to ensure that a mesh that achieves the expected accuracy is obtained. When the refinement conditions are not met, displacement optimization is performed again along the normal direction based on the objective function, and a new backup mesh is iteratively generated. This closed-loop feedback mechanism allows the mesh to gradually approach the optimal solution, which is beneficial for continuously improving the geometric accuracy and texture consistency of the mesh. When the refinement conditions are met, the backup mesh is determined as the refined mesh, which is beneficial for providing a high-quality, high-fidelity facial mesh foundation for subsequent applications, such as ensuring that subsequent rendering, animation, or identity recognition operations based on this refined mesh have higher accuracy and realism.

[0132] In this optional embodiment, as an optional implementation, the evaluation results include at least one of the following: the magnitude of the objective function decrease, the magnitude of the cross-camera consistency term improvement, the reprojection error, the change in vertex normal displacement, and the current iteration number. Furthermore, the specific method by which the back-projection verification module 305 determines whether the current conditions of the backup face mesh meet the preset refinement conditions based on the evaluation results includes: Determine whether the decrease in the objective function is less than a preset decrease, denoted as the first condition; or, Determine whether the improvement in cross-camera consistency is less than a preset improvement, and record this as the second condition; or... Determine whether the reprojection error is less than the preset error, denoted as the third condition; or... Determine whether the change in the normal displacement of the vertex is less than a preset change, denoted as the fourth condition; or... Determine whether the current iteration count has reached the preset maximum iteration count, denoted as the fifth condition; Specifically, when at least one of the first, second, third, fourth, and fifth conditions is met, it is determined that the current condition of the backup facial mesh meets the preset refinement condition.

[0133] As can be seen, this optional implementation can determine whether the mesh meets the refinement conditions based on at least one evaluation index, such as the decrease in the objective function of the updated mesh, the improvement in the cross-camera consistency term, the reprojection error, the change in vertex normal displacement, and the current iteration number. It can improve the flexibility, diversity, and accuracy of the judgment on whether the mesh meets the refinement conditions based on diverse evaluation indexes, reduce the occurrence of delays in determining whether the mesh has not been refined due to errors in the evaluation of a single index, and help improve the accuracy, efficiency, and reliability of mesh refinement.

[0134] Example 4 Please see Figure 6 , Figure 6 This is a schematic diagram of another mesh refinement device based on texture reliability and partition normal displacement disclosed in an embodiment of the present invention. Figure 6 As shown, the mesh refinement device based on texture confidence and partition normal displacement may include: Memory 401 storing executable program code; Processor 402 coupled to memory 401; The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the mesh refinement method based on texture confidence and partition normal displacement described in Embodiment 1 or Embodiment 2 of the present invention.

[0135] Example 5 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the mesh refinement method based on texture reliability and partition normal displacement described in Embodiment 1 or Embodiment 2 of this invention.

[0136] Example 6 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the mesh refinement method based on texture confidence and partition normal displacement described in Embodiment 1 or Embodiment 2.

[0137] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0139] Finally, it should be noted that the mesh refinement method and apparatus based on texture reliability and partition normal displacement disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mesh refinement method based on texture reliability and partitioned normal displacement, characterized in that, The method includes: The credibility decomposition is performed on the texture image of the scanned face acquired by each of the multiple cameras to obtain the decomposition result of that camera. The decomposition result includes at least one of the base reflection layer, high frequency detail layer, residual layer and anomalous response layer. Based on the decomposition results of all the cameras and the pre-determined multiple influence data construction algorithm, a composite texture confidence map is generated; Based on the composite texture confidence map and semantic region information, the initial facial mesh of the scanned face is partitioned to obtain a facial partitioning result. The partitioning result includes at least one of a refinement region, a boundary-restricted region, a preservation region, and an anomaly shielding region. For the vertices of the mesh to be updated determined from the facial partitioning results, the displacement of the vertices of the mesh to be updated is optimized along the normal direction based on a pre-determined objective function to update the initial facial mesh and obtain a backup facial mesh; A re-projection verification operation is performed on the backup facial mesh to determine the refined mesh that meets the preset refinement conditions.

2. The mesh refinement method based on texture reliability and partitioned normal displacement according to claim 1, characterized in that, The step of generating a composite texture confidence map based on the decomposition results of all the cameras and a pre-determined algorithm for constructing multiple influence data includes: The initial facial mesh is projected onto the texture images of all the cameras to obtain camera projection data for one or more key points. The key points include the mesh vertices of the initial facial mesh or the texture sampling points of the texture image. The camera projection data for each key point includes the visible camera corresponding to each key point and the projection position of each key point on the corresponding visible camera. Based on the camera projection data of each key point and the pre-determined multi-influence data construction algorithm, the multi-influence data of each key point is calculated; and based on the multi-influence data of all key points, a composite texture confidence map is generated.

3. The mesh refinement method based on texture reliability and partitioned normal displacement according to claim 2, characterized in that, The step of calculating multiple influence data for each key point based on camera projection data and a pre-determined multiple influence data construction algorithm includes: Based on the projection position of each key point on the corresponding visible camera, the high-frequency detail layer corresponding to the visible camera, and the predetermined cross-camera consistency term construction algorithm, the cross-camera consistency term construction operation is performed on the key point to obtain the cross-camera consistency term data of the key point. Based on the projection position of each key point on the corresponding visible camera, the residual layer and anomaly response layer corresponding to the visible camera, and the pre-determined anomaly penalty term construction algorithm, anomaly penalty term construction operation is performed on the key point to obtain the anomaly penalty term data of the key point. Based on the texture image of the visible camera corresponding to each key point, the base reflection layer of the visible camera, and the pre-determined imaging quality term construction algorithm, an imaging quality term construction operation is performed on the key point to obtain the imaging quality term data of the key point. Based on the projection position of each key point on the corresponding visible camera, the pre-determined reference position of the key point, and the pre-determined reprojection error term construction algorithm, the reprojection error term construction operation is performed on the key point to obtain the reprojection error term data of the key point. Based on the preset semantic region label of each key point and the pre-determined region prior mapping function algorithm, a region prior construction operation is performed on the key point to obtain the region prior data of the key point; The multiple impact data for each key point include cross-camera consistency data, anomaly penalty data, image quality data, reprojection error data, and region prior data.

4. The mesh refinement method based on texture reliability and partition normal displacement according to any one of claims 1-3, characterized in that, The step of optimizing the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function to update the initial facial mesh and obtain a backup facial mesh includes: Obtain the position of the vertex in the mesh to be updated before the update; Based on the predetermined objective function and the displacement range constraint conditions corresponding to the objective function, the required normal displacement of the vertex to be updated is determined according to the vertex position before the update of the vertex to be updated. For the vertex position before the update of the vertex to be updated, the displacement is optimized along the normal direction according to the normal displacement required by the vertex to be updated, to obtain the vertex position after the update of the vertex to be updated. The spare face mesh includes the vertex position after the update of the vertex to be updated and the vertex positions before the update of the remaining vertices of all the vertices except the vertex to be updated.

5. The mesh refinement method based on texture reliability and partition normal displacement according to claim 4, characterized in that, The method further includes: Based on the weight control mechanism corresponding to the predetermined objective function, the region to which the vertex to be updated belongs is determined according to the vertex position before the update of the vertex to be updated. The vertex to be updated whose region is a preset abnormal region is masked to obtain the target vertex whose region is a preset updatable region. Furthermore, the step of optimizing the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function to update the initial facial mesh and obtain a backup facial mesh includes: Based on a predetermined objective function, the target mesh vertices are displacement optimized along the normal direction to update the initial face mesh and obtain a backup face mesh.

6. The mesh refinement method based on texture reliability and partitioned normal displacement according to any one of claims 1, 2, 3, and 5, characterized in that, The step of performing a re-projection verification operation on the backup facial mesh to determine the refined mesh that meets the preset refinement conditions includes: The alternative facial mesh is projected onto the texture images of all the cameras to evaluate the alternative facial mesh and obtain the evaluation result; Based on the evaluation results, determine whether the current conditions of the backup facial mesh meet the preset refinement conditions; When it is determined that the current conditions of the backup face mesh do not meet the refinement conditions, the operation of re-executing the pre-determined objective function to optimize the displacement of the vertices of the mesh to be updated along the normal direction to update the initial face mesh and obtain the backup face mesh is performed. When it is determined that the current conditions of the backup facial mesh meet the refinement conditions, the backup facial mesh is identified as a refinement mesh that meets the preset refinement conditions.

7. The mesh refinement method based on texture reliability and partition normal displacement according to claim 6, characterized in that, The evaluation results include at least one of the following: the magnitude of the decrease in the objective function, the magnitude of the improvement in the cross-camera consistency term, the reprojection error, the change in vertex normal displacement, and the current iteration number; And, the step of determining whether the current conditions of the backup facial mesh meet the preset refinement conditions based on the evaluation results includes: Determine whether the decrease in the objective function is less than a preset decrease, denoted as the first condition; or, Determine whether the improvement in the cross-camera consistency term is less than a preset improvement, and record this as the second condition; or, Determine whether the reprojection error is less than a preset error, and record this as the third condition; or, Determine whether the change in the normal displacement of the vertex is less than a preset change, denoted as the fourth condition; or, Determine whether the current iteration count has reached the preset maximum iteration count, denoted as the fifth condition; Specifically, when at least one of the first condition, the second condition, the third condition, the fourth condition, and the fifth condition is satisfied, it is determined that the current condition of the backup facial mesh meets the preset refinement condition.

8. A mesh refinement device based on texture reliability and partitioned normal displacement, characterized in that, The device includes: The texture confidence decomposition module is used to perform confidence decomposition on the texture image of the scanned face acquired by each of the multiple cameras to obtain the decomposition result of that camera. The decomposition result includes at least one of the base reflection layer, high frequency detail layer, residual layer and anomaly response layer. The composite credibility generation module is used to generate a composite texture credibility map based on the decomposition results of all the cameras and a pre-determined multiple influence data. The face partitioning control module is used to partition the initial face mesh of the scanned face based on the composite texture confidence map and semantic region information to obtain a face partitioning result. The partitioning result includes at least one of a refinement area, a boundary-restricted area, a preservation area, and an anomaly shielding area. The normal displacement optimization module is used to optimize the displacement of the vertices of the mesh to be updated along the normal direction based on a pre-determined objective function, based on the facial partitioning results, so as to update the initial facial mesh and obtain a backup facial mesh. The re-projection verification module is used to perform a re-projection verification operation on the backup facial mesh to determine the refined mesh that meets the preset refinement conditions.

9. A mesh refinement device based on texture reliability and partitioned normal displacement, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the mesh refinement method based on texture confidence and partition normal displacement as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the mesh refinement method based on texture confidence and partition normal displacement as described in any one of claims 1-7.

Images