Three-dimensional face high-fidelity texture generation method based on multi-view high-definition image

Abstract

The invention relates to a three-dimensional face high-fidelity texture generation method based on a multi-view high-definition image, and the method comprises the steps: constructing a three-dimensional face triangular mesh model through employing a camera fusion weight and an external parameter based on the cosine square of a view angle, and solving a texture mixed weight of each camera to a mesh triangular patch; the cylindrical UV expansion method is used for adapting to the geometrical characteristics of the human face, so that the texture stretching distortion can be obviously reduced; high-frequency detail reservation and low-frequency hue smooth transition are achieved through a frequency-band-divided texture fusion strategy, a high-quality texture source is optimized in cooperation with view angle cosine square weight and depth shielding detection, and fusion joints and detail dilution are avoided.

Description

Technical Field

[0001] This invention relates to a method for generating high-fidelity facial textures, and more particularly to a method for generating three-dimensional high-fidelity facial textures based on multi-view high-definition images. Background Technology

[0002] High-fidelity 3D face generation is a core technology in fields such as film and television special effects, virtual digital humans, and medical assistance. Its core requirement is to recreate the microscopic texture details of a face and ensure the integrity and consistency of the texture through multi-view image acquisition and processing. Multi-camera synchronous acquisition schemes have become the mainstream technical approach in this field due to their wide field of view coverage and high data acquisition efficiency.

[0003] While existing technologies have yielded some improvements in texture generation quality, significant shortcomings remain. Currently used techniques such as sRGB linearization and cross-camera brightness normalization rely on global region-based brightness statistics, making them susceptible to background interference. Furthermore, they lack a frequency-division fusion strategy, making it difficult to balance detail sharpness and tonal consistency. Additionally, they do not adapt to the cylindrical geometry of human faces and do not provide a dedicated texture hole repair solution, failing to meet the specific requirements for high-fidelity facial texture generation.

[0004] While existing technologies cover fundamental aspects such as multi-camera usage and texture fusion, they still have shortcomings in face-specific optimization: First, multi-camera pose calibration lacks a dedicated calibration combination centered on a reference camera, leading to accumulated pose correlation errors. Second, texture unfolding often uses planar or spherical parameterization, easily causing stretching and distortion of textures in facial cheeks and forehead areas. Third, the calculation of fusion weights does not fully consider the impact of viewing angle on texture quality; equal weights or simple weighting methods cannot optimize high-quality textures from the positive viewing angle. Fourth, there is a lack of precise and effective techniques for repairing voids in facial textures, affecting texture integrity. These problems make it difficult for existing technologies to meet the requirements of high-fidelity applications in terms of detail preservation, distortion control, and integrity of facial textures, necessitating a targeted technical solution.

[0005] In view of the above-mentioned shortcomings, the designer actively researched and innovated in order to create a method for generating high-fidelity textures of three-dimensional faces based on multi-view high-definition images, making it more valuable for industrial applications. Summary of the Invention

[0006] To address the aforementioned technical problems, the purpose of this invention is to provide a method for generating high-fidelity textures of three-dimensional human faces based on multi-view high-definition images.

[0007] The present invention provides a method for generating high-fidelity textures of three-dimensional faces based on multi-view high-definition images, which includes the following steps: Step 1: Perform multi-camera pose calibration, sequentially complete single-camera intrinsic parameter calibration and pairwise camera extrinsic parameter calibration, and output the camera intrinsic parameter matrix, distortion coefficients and extrinsic parameter transformation matrix between cameras; Step 2: Reconstruct the face mesh model based on depth data. The depth data synchronously acquired by multiple structured light cameras is used as input. The point cloud coarse registration and fine registration are completed and fused by combining the extrinsic transformation matrix of Step 1. After mesh reconstruction and optimization, a topologically complete 3D face triangular mesh model is output. Step 3: Perform cylindrical UV texture unwrapping on the 3D face triangular mesh model from Step 2, mapping the 3D coordinates of the model vertices to the 2D UV texture space, and outputting the UV coordinate pairs of each vertex. Step 4: Perform preprocessing on the high-definition images acquired by multiple cameras, sequentially completing sRGB image linearization and cross-camera brightness adjustment based on the image center region, and outputting a group of multi-camera linear images with consistent brightness; Step 5: Calculate the camera fusion weights based on the square of the viewpoint cosine. Combine the camera intrinsic and extrinsic parameter calibrations from Step 1 with the 3D face triangular mesh model from Step 2 to obtain the texture blending weights of each camera on the mesh triangular facets. Step 6: Perform texture space rasterization and multi-camera back-projection sampling. In sequence, perform texture space bounding box rasterization, pixel-by-pixel centroid coordinate calculation, and multi-camera back-projection color sampling with validity filtering. Output the independent texture buffer and weight map of each camera. Step 7: Perform multi-camera texture fusion based on Laplacian and Gaussian pyramids, sequentially completing the construction of Laplacian and Gaussian pyramids, differential fusion of optimal camera exclusive in high-frequency layers and weighted average in low-frequency layers, pyramid reconstruction, and outputting the fused linear texture image. Step 8: Perform post-processing on the fused linear texture image from Step 7, sequentially completing sRGB linear space transformation and filling internal texture holes based on the Navier-Stokes equation, outputting a detailed, seamless, and hole-free 3D high-fidelity texture map of the human face.

[0008] Furthermore, in the above-mentioned method for generating high-fidelity textures of 3D faces based on multi-view high-definition images, the single-camera intrinsic parameter calibration in step 1 adopts the Zhang Zhengyou planar target calibration method. The checkerboard calibration board is placed within the camera's field of view, and 20 calibration images from different angles are acquired. The pixel coordinates of the checkerboard corner points in each image are extracted, and their correspondence with the 3D coordinates in the physical coordinate system of the calibration board is established. The initial values ​​of the camera intrinsic parameters are obtained based on homography matrix decomposition. A distortion model constructed by radial and tangential directions is introduced. Through a nonlinear minimization algorithm (Levenberg-Marquardt algorithm), the intrinsic parameters and distortion coefficients are jointly optimized to minimize the reprojection error within the maximum likelihood estimation framework, and the camera intrinsic parameter matrix and distortion coefficients are output.

[0009] Furthermore, in the aforementioned method for generating high-fidelity 3D face textures based on multi-view high-definition images, step 1, the pairwise camera extrinsic parameter calibration step, involves using the physical corner coordinates of a checkerboard calibration board as a 3D reference point in the world coordinate system. The calibration board is placed within the common field of view of the two cameras, and at least 10 sets of synchronized images are acquired. The 2D pixel coordinates of the checkerboard corner points in each image are extracted to form sub-pixels, and a point correspondence between them and the 3D coordinates in the world coordinate system is established. Based on the known camera intrinsic parameter matrix, the camera pose is estimated using a PnP solver, and the rotation matrix and translation vector are optimized by minimizing the reprojection error of the 3D points on the image plane. The solution results are weighted and averaged according to the reprojection error, and image pairs with errors exceeding the threshold are removed before re-estimation. The relative rotation matrix and translation vector between the two cameras are output, forming the extrinsic parameter transformation matrix between the cameras.

[0010] Furthermore, in the above-mentioned method for generating high-fidelity textures of 3D faces based on multi-view high-definition images, step 2, the face mesh model reconstruction is specifically divided into the following steps: performing depth range filtering on the original point clouds output by each camera; applying rigid body transformation to the multi-path point clouds using the extrinsic transformation matrix from step 1 and unifying them to the central camera coordinate system to complete coarse registration; performing fine registration on the multi-path point clouds using the ICP iterative nearest point algorithm and merging them into a unified point cloud; sequentially performing filtering, smoothing, and surface projection on the fused point cloud and estimating the normals of each point; generating a closed triangular mesh using the Poisson surface reconstruction algorithm; and outputting a topologically complete 3D face triangular mesh model after mesh optimization.

[0011] Furthermore, in the above-mentioned method for generating high-fidelity textures of 3D faces based on multi-view high-definition images, step 3, the cylindrical UV texture unfolding is specifically divided into the following steps: establishing a coordinate system with the X-axis as the horizontal direction, the Y-axis as the vertical direction, and the Z-axis as the direction of the nose tip; calculating the azimuth angle φ=arctan2(X,Z) of each vertex of the 3D face triangular mesh model on the XZ plane to determine the horizontal UV coordinate u; determining the vertical UV coordinate v by the normalized position of the vertex Y value relative to the vertical range of the model; performing automatic interval stretching and linear mapping to [0,1] on the coordinate u; and outputting the UV coordinate pairs of each vertex.

[0012] Furthermore, in the above-mentioned method for generating high-fidelity textures of 3D faces based on multi-view high-definition images, the high-definition image preprocessing in step 4 specifically includes the following steps: linearizing the sRGB image, normalizing the pixel values ​​of the sRGB encoding format to a floating-point range of 0-1; according to the sRGB standard gamma-reduction transformation rules, applying a linear proportional relationship transformation to pixels whose normalized pixel values ​​do not exceed a preset low value threshold, and applying a nonlinear transformation transformation in the form of a power function to pixels whose values ​​exceed the preset low value threshold, to obtain a linear brightness value that is proportional to the physical illumination. The cross-camera brightness adjustment involves extracting the central region of each camera image with a height and width margin of 20% as a statistical region, performing validity filtering on the pixels within the statistical region and excluding dark pixels, calculating the average brightness of each color channel, taking the arithmetic mean of the average brightness of each color channel of all cameras as the global target brightness, calculating the brightness adjustment gain of each camera image by channel, and performing a limiting process with a lower limit of 0.85 times and an upper limit of 1.15 times on the gain, multiplying the limiting adjustment coefficient pixel by pixel onto the linear image of the corresponding camera and performing truncation processing to obtain a multi-camera linear image group with consistent brightness.

[0013] Furthermore, in the above-mentioned method for generating high-fidelity textures of 3D faces based on multi-view high-definition images, step 5, specifically the calculation of camera fusion weights based on the square of the viewpoint cosine, involves: transforming the vertices of the 3D mesh from the world coordinate system to the camera coordinate system using the camera extrinsic matrix; projecting them onto the pixel plane using the intrinsic matrix; calculating the center coordinates corresponding to the mean of the three vertices of each triangular facet in the 3D mesh of the 3D face triangular mesh model; obtaining the unit viewing direction vector between the camera optical center and the facet center; taking the dot product of the facet unit normal vector and the unit viewing direction vector to obtain the viewpoint cosine value and truncating the negative value to zero; and taking the square of the truncated cosine value as the texture fusion weight of the camera for that facet.

[0014] Furthermore, in the above-mentioned method for generating high-fidelity textures of 3D faces based on multi-view high-definition images, in step 6, the texture space rasterization and multi-camera back-projection sampling are specifically divided into the following steps: texture space bounding box rasterization, obtaining the UV coordinates of the three vertices of the triangular facets of the 3D facet mesh model, determining the axis-aligned bounding box of the facet in the texture space and converting it into a range of pixel integer coordinates, cropping it to the boundary of the texture image as the rasterization area, and skipping the degenerate facets; For each texture pixel within the bounding box, the normalized texture coordinates of its pixel center are used as the target point. The centroid weight is calculated using the centroid coordinate formula. Only when all centroid weights are greater than or equal to 0 are they determined to be valid pixels inside the patch. Multi-camera back-projection color sampling is performed. For effective pixels, the back-projection coordinates in the camera image are obtained by interpolation of the centroid coordinates. The validity filtering of the three vertex depth values ​​of the facets is performed within the effective size range of the forward and back-projection coordinates. Colors are sampled from the linearized image and the viewpoint cosine weights are written into the weight map. The independent texture buffers and weight maps of each camera are output.

[0015] Furthermore, in the above-mentioned method for generating high-fidelity textures of 3D faces based on multi-view high-definition images, in step 7, the multi-camera texture fusion of frequency bands is specifically divided into the following steps: constructing a Laplacian pyramid for the texture image of each camera by repeatedly downsampling and subtracting from upsampling; and constructing a Gaussian weight pyramid by synchronously and repeatedly downsampling the normalized weight map of each camera. For each high-frequency layer of the Laplacian pyramid, the camera corresponding to the maximum weight for each pixel is selected, and that camera exclusively receives the high-frequency detail contribution for that pixel, while the high-frequency contributions of other cameras are set to zero. For the low-frequency tonal image at the top layer of the Laplacian pyramid, a weighted fusion strategy based on normalized weights is adopted. The fusion process is as follows: Suppose there are N camera inputs. For the top layer (low-frequency layer) of the Laplacian pyramid, the fused low-frequency image... Calculated using the following formula: .in: Let represent the normalized fusion weight of the i-th camera in this layer, and satisfy the normalization constraint condition. , This represents the low-frequency tone image of the top layer of the Laplacian pyramid corresponding to the i-th camera; The layered fusion results are combined into a fused Laplacian pyramid. Starting from the top low-frequency image, the layers are upsampled layer by layer and the corresponding high-frequency residual layers are superimposed to restore the original texture resolution and truncate the pixel values ​​to the legal range, outputting the fused linear texture image.

[0016] Furthermore, in the above-mentioned method for generating high-fidelity textures of 3D faces based on multi-view high-definition images, in step 8, the texture post-processing is specifically divided into the following steps: sRGB linear space conversion, using a piecewise function according to the sRGB standard specification to convert the linear texture image into sRGB encoding space, quantizing the conversion result into an 8-bit unsigned integer format, and flipping the texture image and mesh overlay mask along the vertical axis. To fill the holes inside the texture, the weight maps of each camera are summed pixel by pixel to obtain the total weight map. The pixel regions with zero weights within the mesh overlay mask are extracted as hole masks. An image inpainting algorithm based on the Navier-Stokes equations is used to fill the hole regions inward with the known texture pixels around the holes as references, restoring the visual continuity of the texture and outputting a high-fidelity 3D face texture map.

[0017] By means of the above-described solution, the present invention has at least the following advantages: 1. Strong hardware configuration adaptability and high acquisition accuracy. The multi-camera synchronous acquisition system, which combines RGBD and RGB cameras, and is equipped with a synchronous trigger module to achieve precise data synchronization, has an acquisition resolution of no less than 2K, which can fully preserve the microscopic texture details of human faces such as pores and wrinkles, laying a data foundation for high-fidelity facial data generation.

[0018] 2. High pose calibration accuracy and strong anti-interference capability. The single-camera intrinsic parameter calibration combines sub-pixel corner detection and iterative optimization algorithms, while the extrinsic parameter calibration achieves accurate pose solving through synchronous image acquisition and abnormal data removal, effectively reducing noise interference and providing reliable spatial parameter support for subsequent point cloud registration and texture sampling.

[0019] 3. High-quality texture generation with minimal distortion and seams. A cylindrical UV unwrapping method is employed to adapt to facial geometry, significantly reducing texture stretching distortion. A frequency-segmented texture fusion strategy preserves high-frequency details while ensuring smooth transitions in low-frequency tones. Combined with viewpoint cosine square weighting and depth occlusion detection, high-quality texture sources are prioritized to avoid fusion seams and detail dilution.

[0020] 4. Excellent texture integrity and visual continuity: The texture hole is accurately located by weight map, and the hole is naturally filled by the image inpainting algorithm of Navier-Stokes equation. At the same time, after sRGB space conversion and orientation correction, the final output texture map has no holes and no seams, and has a high degree of fit with the 3D face model.

[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0022] Figure 1 This is a schematic diagram showing the distribution of 6 synchronously acquiring cameras facing the face.

[0023] Figure 2 This is a schematic diagram of the reconstructed face mesh model.

[0024] Figure 3 This is a schematic diagram illustrating the generation of high-fidelity textures for a 3D human face. Detailed Implementation

[0025] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0026] like Figures 1 to 3This paper presents a method for generating high-fidelity 3D facial textures based on multi-view high-definition images. Its unique feature lies in its hardware acquisition system, which comprises six multi-structured light cameras: three RGBD cameras (RGBD_Left, RGBD_Center, and RGBD_Right) and three RGB cameras (RGB_Left, RGB_Center, and RGB_Right). Using RGBD_Center as the reference camera, the spatial relative positions of all cameras are fixed, and a synchronization trigger module is configured to achieve synchronous acquisition of RGB images and depth data. The camera acquisition resolution is no less than 2K, ensuring the accuracy of capturing facial micro-textures such as pores and wrinkles. Furthermore, all algorithms are implemented on a computing terminal equipped with a multi-core processor and an independent graphics processing unit, supporting large-scale matrix operations and parallel computing, meeting the efficiency and accuracy requirements of algorithm execution. The specific implementation includes the following steps.

[0027] Step 1, Multi-camera Pose Calibration. Its core is to calibrate the internal optical parameters of a single camera and determine the relative spatial pose relationships between multiple cameras. The final output includes the camera intrinsic parameter matrix, distortion coefficients, and extrinsic parameter transformation matrix between cameras, providing accurate mathematical basis for subsequent point cloud registration and texture projection sampling. This step sequentially completes the calibration of single-camera intrinsic parameters and the calibration of extrinsic parameters between pairs of cameras. Both calibrations use a high-precision checkerboard calibration board as the reference target. The calibration board uses a black and white alternating square grid, with known fixed values ​​for the physical dimensions of the grid points, and the flatness error of the calibration board is ≤0.01mm.

[0028] 1.1 Single-camera intrinsic parameter calibration employs Zhang Zhengyou's planar target calibration method. A checkerboard calibration board is placed within the complete field of view of the corresponding camera. By adjusting the pose of the calibration board, including translation and rotation, 20 calibration images are acquired at different angles and distances. Simultaneously, the calibration images must cover the center and edge regions of the camera's field of view, and the calibration board should not be severely tilted, obstructed, or blurred in the images. The number of images is selected based on the optimal implementation specifications of Zhang Zhengyou's calibration method. 20 calibration images effectively improve the accuracy of solving the initial intrinsic parameter values. For each calibration image, a sub-pixel-level corner detection algorithm is used to extract the pixel coordinates of the checkerboard corners. The sub-pixel detection accuracy can reach 0.1 pixels. A one-to-one correspondence is established between the extracted corner pixel coordinates and the three-dimensional coordinates in the calibration board's physical coordinate system. The calibration board's physical coordinate system has the first corner point at the lower left corner as the origin, the calibration board plane as the XY plane, and the Z-axis perpendicular to the calibration board plane.

[0029] Based on the above point correspondence, the initial values ​​of the camera intrinsic parameters are obtained through homography matrix decomposition. The camera intrinsic parameter matrix is ​​a 3×3 fixed-format matrix, in the form of: Where fx and fy are the focal lengths of the camera in the x and y directions (in pixels), respectively, and cx and cy are the coordinates of the principal point in the x and y directions, i.e., the intersection of the optical axis and the image plane (usually near the image center). Due to optical distortion in the camera lens, a distortion model constructed from radial and tangential distortion is introduced. The distortion coefficient vector is a 1×5 vector, in the form of: Where k1, k2, and k3 are the first-order, second-order, and third-order radial distortion coefficients, respectively, and p1 and p2 are the tangential distortion coefficients.

[0030] With minimizing the reprojection error as the optimization objective, the Levenberg-Marquardt algorithm is used to jointly optimize the intrinsic parameter matrix and distortion coefficients within the maximum likelihood estimation framework. The objective function for optimization is: Where mij is the measured pixel coordinate of the j-th corner point in the i-th image. Let Ri and ti be the theoretical reprojection coordinates, respectively, and let Mj be the rotation matrix and translation vector corresponding to the i-th image. Mj represents the 3D physical coordinates of the j-th corner point, N is the number of calibration images, and M is the number of corner points in a single image. The iteration stops when the optimization algorithm converges, and the intrinsic parameter matrix and distortion coefficients of the camera are output. In practice, the Camera Calibrator tool in Matlab software is used to sequentially complete the intrinsic parameter calibration of six cameras using the above method, obtaining a unique intrinsic parameter matrix and distortion coefficients for each camera. This provides accurate internal parameter data for subsequent pose estimation, 3D point cloud projection, and texture backprojection sampling.

[0031] 1.2 Pairwise Camera Extrinsic Parameter Calibration. The core of pairwise camera extrinsic parameter calibration is to establish the relative spatial pose relationship between each camera and the reference camera RGBD_Center, outputting the rotation matrix R and translation vector t between each pair of cameras to form the extrinsic parameter transformation matrix, thereby achieving the unification of multi-camera data in the same coordinate system. This invention addresses the shortcomings of existing extrinsic parameter calibration results, such as significant noise interference and ineffective screening of abnormal image pairs. It improves the accuracy of extrinsic parameter calibration through synchronous image pair acquisition, sub-pixel corner detection, reprojection error optimization, and abnormal image pair removal. The specific implementation process is as follows: Using the physical corner coordinates of the checkerboard calibration board as the three-dimensional reference points in the world coordinate system (the world coordinate system is consistent with the physical coordinate system of the calibration board), the calibration board is placed within the common field of view of the two cameras to be calibrated, ensuring that the calibration board is clearly visible in the images of both cameras without serious obstruction. At least 10 sets of synchronized image pairs are acquired through the camera synchronization trigger module. Synchronous acquisition ensures that the two cameras image the calibration board at the same time, avoiding errors in point correspondence caused by slight movement of the calibration board.

[0032] For each pair of synchronized images, a sub-pixel corner detection algorithm is used to extract the two-dimensional pixel coordinates of the checkerboard corners in both images, establishing a point correspondence between these two-dimensional pixel coordinates and their three-dimensional coordinates in the world coordinate system. Based on the known intrinsic parameter matrices of the two cameras obtained in step 1.1, the EPnP solver is used to independently estimate the camera pose for each image. (The EPnP solver is an algorithm for solving the perspective-n-point problem, which can quickly and accurately solve for the initial camera pose.) With the objective of minimizing the reprojection error of 3D points on the image plane, the rotation matrix and translation vector corresponding to the camera pose are optimized. The objective function for optimization is: .

[0033] Where ui is the measured pixel coordinate of the i-th corner point. Let K be the perspective projection function, K be the camera intrinsic parameter matrix, Mi be the corresponding 3D coordinates in the world coordinate system, and N be the number of corner points. The rotation matrix is ​​a 3×3 orthogonal matrix of the form: Its elements all take values ​​between [-1, 1]. The three columns correspond to the direction vectors of the x, y, and z axes of camera A in camera B coordinate system; the translation vector is a 3×1 column vector, in the form of: The three components correspond to the displacements in the x, y, and z directions, respectively, and are expressed in millimeters (mm).

[0034] The solution results for all image pairs are processed by weighted average of reprojection error. Image pairs with smaller reprojection errors are assigned higher weights. At the same time, a reprojection error threshold is set (the threshold is at the pixel level, set to 2 pixels) to remove invalid image pairs with errors exceeding the threshold. The camera pose is re-estimated for the remaining valid image pairs, and the relative rotation matrix R and translation vector t between the two sets of cameras are finally output. R and t constitute the extrinsic transformation matrix between the cameras.

[0035] In practice, pairwise extrinsic parameter calibration is performed for five specific camera pairs: RGBD_Left and RGBD_Center, RGBD_Right and RGBD_Center, RGB_Center and RGBD_Center, RGB_Left and RGBD_Left, and RGB_Right and RGBD_Right. The calibration of all camera pairs is based on RGBD_Center as the final reference. Through the above five calibration results, rigid body transformation of all acquired data from the six cameras to the RGBD_Center coordinate system can be achieved, providing a precise spatial pose transformation basis for subsequent multi-camera point cloud registration and multi-view texture projection.

[0036] Step 2: Reconstruction of the face mesh model based on depth data. This step uses RGB-D depth data simultaneously acquired by six cameras as the core input, and utilizes depth point cloud data output from three RGBD cameras, combined with the extrinsic transformation matrix between cameras obtained in Step 1, to reconstruct a topologically complete 3D face triangular mesh model from the original point cloud. This model serves as the 3D geometric carrier for all subsequent texture generation operations; all texture sampling and fusion revolve around this model. High-quality 3D face triangular mesh model reconstruction can be achieved through adaptive ICP search radius, multi-strategy point cloud optimization, multi-round normal correction, and refined mesh optimization. The specific implementation process sequentially completes seven sub-processes: point cloud depth range filtering, coarse point cloud registration, fine point cloud registration, fused point cloud optimization, point cloud normal estimation and correction, Poisson surface reconstruction, and mesh optimization, detailed below.

[0037] 2.1 Point Cloud Depth Range Filtering. The raw depth point clouds output by each RGBD camera are filtered, and an effective range of depth values ​​is set based on the actual working distance of face acquisition. During implementation, the effective range is selected from the leading edge of the RGB_Center to the key point of the philtrum, typically 200-220mm. Optimal, 210mm can be selected as the default preset value. This removes point cloud data with depth values ​​exceeding this range; this data represents near-field noise or far-field background and is not valid point cloud data for the face. Filtering can initially remove irrelevant point cloud interference.

[0038] 2.2 Coarse Point Cloud Registration. Using the extrinsic transformation matrix between cameras obtained in step 1, a rigid body transformation is applied to the depth point clouds of RGBD_Left and RGBD_Right, changing only the spatial position and orientation of the point clouds without altering their shape and size. Then, the two point clouds are unified into the coordinate system of RGBD_Center, completing coarse point cloud registration and initially aligning the three RGBD point clouds in space, laying the foundation for fine registration.

[0039] 2.3 Fine Point Cloud Registration. The adjacent overlapping regions of the three point clouds after coarse registration are extracted. This region is the core area for fine point cloud registration. The search radius of the ICP iterative nearest point algorithm is adaptively determined based on the point cloud density. Unlike the fixed search radius of existing technologies, the adaptive search radius can be dynamically adjusted according to the neighborhood point density of the current point cloud region. This allows the ICP algorithm to use a small radius in dense point cloud regions such as the eyes and nose, and a large radius in sparse point cloud regions such as the cheeks, improving registration accuracy. Fine registration is performed on the three point clouds using the ICP algorithm, repeatedly finding corresponding point pairs, calculating the transformation matrix, and minimizing the Euclidean distance between point pairs. When the number of iterations reaches a preset value (default 50) or the average Euclidean distance between point pairs is less than the convergence threshold (default 0.1mm), the iteration stops, and the three point clouds are merged into a unified point cloud.

[0040] 2.4 Optimization of Merged Point Clouds. Multi-strategy optimization is performed on the merged point cloud, sequentially performing statistical filtering, radius filtering, voxel downsampling, Laplacian smoothing, and moving least squares surface projection. Statistical filtering removes isolated noise points from the point cloud; radius filtering removes low-density point cloud regions; voxel downsampling reduces the number of points while maintaining the point cloud shape to improve subsequent processing efficiency; Laplacian smoothing eliminates local spikes in the point cloud; and moving least squares surface projection makes the point cloud more closely resemble the natural curved surface of a human face, ultimately resulting in a high-quality merged point cloud with uniform density and no noise.

[0041] 2.5 Point Cloud Normal Estimation and Correction. The normal of each point cloud point is estimated based on the KD-tree neighborhood search algorithm. The KD-tree can quickly find the neighborhood set of each point, and the initial value of the normal is obtained through the covariance matrix decomposition of the neighborhood set. Since directly estimated normals suffer from inconsistencies in direction and local flips, the following steps are performed sequentially: tangent plane consistency propagation, flip cluster detection and correction, inconsistency point removal, and multi-round iterative normal smoothing. Finally, a smooth and continuous directed normal field is constructed, providing accurate normal information for subsequent surface reconstruction.

[0042] 2.6 Poisson Surface Reconstruction. Using a fused point cloud with a directed normal field as input, the Poisson surface reconstruction algorithm is employed to solve for the spatial indicator function. This algorithm spatially voxels the directed point cloud, uses the Poisson equation to solve for the implicit surface of the fitted point cloud, extracts the isosurface of the implicit surface, and generates a closed triangular mesh model.

[0043] 2.7 Mesh Optimization. The generated original triangular mesh model undergoes fine-grained optimization, sequentially performing low-density vertex culling, hole closure, Taubin smoothing, and connected component filtering: low-density vertex culling removes redundant vertices from the mesh; hole closure fills in tiny holes in the mesh; Taubin smoothing eliminates local bumps and depressions in the mesh while maintaining its topology; and connected component filtering removes isolated mesh regions that are not connected to the main face.

[0044] Through the above seven sub-processes, the final output is a topologically complete, smooth, and structurally regular 3D face triangular mesh model. This model has no holes, no redundant vertices, and no local bumps, meeting the 3D geometric requirements for subsequent high-fidelity texture generation. The final result is as follows: Figure 2 As shown.

[0045] Step 3: Cylindrical UV Texture Unwrapping. Cylindrical UV texture unwrapping uses the 3D triangular mesh model of the face obtained in Step 2 as input. It precisely maps the 3D coordinates of the model vertices to the 2D UV texture space, establishing a one-to-one correspondence between 3D geometric vertices and 2D texture pixels. Finally, it outputs the UV coordinate pairs of each vertex. These coordinate pairs are the core parameterization basis for subsequent texture sampling, rasterization, and mapping. This invention uses a cylindrical parameterization method to complete the mapping, adapting to the cylindrical morphological features of the face, effectively avoiding texture stretching distortion in areas such as the cheeks and forehead. The specific implementation process is as follows.

[0046] First, a dedicated coordinate system with cylindrical parameterization is established. Using the tip of the nose in the 3D facial mesh model as the reference point, the X-axis is set to the horizontal direction, the Y-axis to the vertical direction, and the Z-axis to the direction the nose tip faces. The origin of the coordinate system is located at the tip of the nose. For each vertex in the 3D facial triangular mesh model, its azimuth angle in the XZ plane is calculated based on its 3D coordinates in this coordinate system. The formula for calculating the azimuth angle is: Where X and Z are the X-axis and Z-axis coordinates of the vertex in this coordinate system, respectively, and the horizontal UV coordinates of the vertex are determined by this azimuth angle. Vertical UV coordinates The position of the vertex Y relative to the normalized position of the model's longitudinal range is then determined.

[0047] For horizontal UV coordinates Automatic interval stretching is performed to linearly map the azimuth range actually occupied by the face mesh to the interval [0,1], perpendicular to the UV coordinates. Similarly, the values ​​are taken in the interval [0,1], and the origin is taken as the maximum value of the Y value in the mesh model (the top of the head position), and the values ​​increase sequentially downwards, ultimately resulting in the horizontal UV coordinates. and vertical UV coordinates The specific calculation formula is as follows: ; .in, Let be the azimuth angle of the i-th vertex. and These are the minimum and maximum azimuth angles of all vertices, respectively. Let be the Y-axis coordinate of the i-th vertex. and These are the minimum and maximum values ​​of the Y-axis coordinates of all vertices, respectively.

[0048] The UV coordinate pairs (u,v) of each vertex in the 3D face triangular mesh model are calculated using the above formula. All UV coordinate pairs of the vertices are stored and output as the basic data for subsequent texture space rasterization and sampling, realizing low-distortion mapping from 3D geometry to the 2D plane.

[0049] Step 4, High-Resolution Image Preprocessing. High-resolution image preprocessing takes high-resolution RGB images simultaneously acquired by six cameras as input, and sequentially performs sRGB image linearization and cross-camera brightness adjustment based on the image center region. This eliminates non-linear color errors and brightness deviations between multiple cameras, ultimately outputting a group of linear images from multiple cameras with consistent brightness. This provides a physically consistent and brightness-uniform image data foundation for subsequent texture fusion.

[0050] 4.1, sRGB image linearization.

[0051] In actual implementation, the acquired high-definition RGB images are stored in sRGB non-linear encoding format. Directly performing color calculations and image fusion in this space introduces non-linear errors, causing the fusion result to deviate from the physical true value. Therefore, it is necessary to linearize the sRGB images, converting them into linear brightness values ​​proportional to physical illumination. This invention achieves this through the following steps: First, the pixel values ​​in sRGB encoding format are normalized from 8-bit unsigned integers to a floating-point range of [0,1]. The normalization formula is as follows: in, These are the normalized floating-point pixel values. It is the original 8-bit integer pixel value, which ranges from 0 to 255.

[0052] Then, the normalized floating-point pixel values ​​are subjected to the following piecewise transformation to obtain linear brightness values ​​that are linearly related to the physical irradiance: The meanings of each symbol and its value are as follows: : Normalized sRGB floating-point pixel value, with a value range of [0,1]; The output linear brightness value, ranging from [0,1], is proportional to the physical irradiance. 0.04045: The segmentation threshold specified by the sRGB standard (IEC61966-2-1) is the dividing point between linear and non-linear segments; extremely dark pixels below the threshold exhibit significant linear perception characteristics and are directly scaled linearly. 1 / 12.92≈0.0773: Scaling factor for the linear segment, used to ensure the continuity of the function between the two segments at the segmentation point; 2.4: The exponent of the nonlinear segment power function corresponds to the physical gamma characteristic of the sRGB monitor, with an effective gamma of approximately 2.2; 1.055, 0.055: Gain and offset coefficients for the nonlinear segment, used to ensure the continuity and smooth transition of the function at the segmentation point.

[0053] Perform linearization processing on all high-definition RGB images captured by the cameras in sequence using the method described above.

[0054] 4.2 Brightness Statistics Based on the Central Region and Cross-Camera Brightness Adjustment. After linearization, the central region of interest (ROI) is extracted from each camera image. Specifically, 20% of the image height and width are used as margins, and the central portion of the image is cropped as the statistical region to eliminate interference from the surrounding background, shadows, and vignetting. Within the cropped central region, pixels are further filtered for effectiveness, excluding pixels with channel mean values ​​below a preset dark pixel threshold to avoid offsetting the brightness estimation results by pure black backgrounds or underexposed areas.

[0055] For the effective pixels that pass the filter, the mean value of each color channel is calculated, and this is used as a representative brightness estimate of the camera image. If the number of effective pixels is zero, the method degenerates into a global mean calculation for the entire image to ensure robustness in extreme cases. After calculating the mean brightness of the central region of all camera images, the arithmetic mean of all camera means across each color channel is taken and determined as the global target brightness for brightness normalization. Subsequently, for each camera image, the scaling factor between its current mean brightness and the target brightness is calculated for each channel, and this scaling factor is used as the brightness adjustment gain for that image.

[0056] To prevent overcorrection due to extreme brightness deviations of individual cameras, the calculated channel gains are limited to a reasonable range of 0.85 to 1.15 times. After gain limiting, the adjustment coefficients are multiplied pixel by pixel onto the corresponding linear image of the camera, and the results are truncated to prevent pixel overflow, ultimately resulting in a multi-camera linear image group with consistent brightness.

[0057] Step 5: Camera fusion weight calculation based on viewpoint cosine. The camera fusion weight calculation based on the square of the viewpoint cosine combines the camera intrinsic and extrinsic parameter calibrations obtained in Step 1 with the 3D face triangular mesh model obtained in Step 2 to calculate the texture blending weight of each camera on the mesh triangular facets. This weight characterizes the contribution of each camera to the texture of different regions of the face, enabling the selection of texture sources from cameras with positive viewpoints and high resolution, avoiding the impact of texture stretching distortion from grazing angle acquisition on the fusion result. In actual implementation, only three RGB cameras—RGB_Left, RGB_Center, and RGB_Right—are selected for the texture fusion weight calculation because their acquired images have higher resolution and are more suitable as texture sources. The specific implementation process is as follows.

[0058] First, using the camera extrinsic transformation matrix obtained in step 1, all vertices of the 3D face triangular mesh model are transformed from the world coordinate system to the camera coordinate system of each RGB camera. Then, the vertices are projected onto the camera's pixel plane using the camera's intrinsic matrix to obtain the two-dimensional pixel coordinates and depth values ​​of each vertex in the corresponding camera image. The two-dimensional pixel coordinates are used for subsequent texture sampling, and the depth values ​​are used for subsequent occlusion detection.

[0059] For each triangular facet in the 3D face triangular mesh model Calculate the center coordinates of its patch. The center coordinates of the triangular facet are the arithmetic mean of the three-dimensional coordinates of the three vertices of the facet. The optical center of each RGB camera... Using the camera's optical center as a reference point, calculate the difference vector between the camera's optical center and the center of the surface patch. Normalize this difference vector to obtain the unit viewing direction vector. The calculation formula is as follows: .in, Let be the magnitude of the difference vector.

[0060] Obtain the unit normal vector of the triangular facet. The unit normal vector of the surface With unit line-of-sight vector Perform a dot product operation to obtain the cosine value of the viewpoint corresponding to the camera for that area, and truncate negative values ​​to zero (backward areas do not contribute to the texture). The calculation formula is as follows: .

[0061] During implementation, for surfaces with a view cosine value greater than 0.01 (i.e., surfaces facing the camera), occlusion detection is further performed to remove surfaces that, although facing the camera, are occluded by other surfaces and cannot be directly observed by the camera, thus achieving depth occlusion detection. The specific implementation of occlusion detection is as follows: A ray is projected into the scene with a small offset (1e−3, consistent with point cloud coordinates, in mm) from the center of the surface along its unit outward normal vector direction. The ray direction is the unit vector pointing from the center of the surface to the camera's optical center. If the following three conditions are met, the surface is determined to be occluded: First, the ray did indeed hit a triangular facet in the scene (i.e., the hit result was not null). Secondly, the piece of dough that was hit was not the dough itself; Third, if the distance from the ray's hit point to the starting point is less than the distance from the center of the patch to the camera's optical center (with a tolerance of 2 units), it indicates that the hit point is located on the path between the patch and the camera. Patches identified as occluded have their weights set to zero in subsequent steps and do not participate in texture contribution.

[0062] After the above processing, for surfaces with a view cosine value greater than 0.01 (i.e., surfaces facing the camera), occlusion detection is further performed to remove surfaces that, although facing the camera, are occluded by other surfaces and cannot be directly observed by the camera. The occlusion detection process is as follows: using a small offset distance from the surface center along its unit outward normal vector as the ray origin, and using the unit vector pointing from the surface center to the camera's optical center as the ray direction, a ray is projected into the ray projection scene. The offset distance is 10. −3 Its unit is consistent with the point cloud coordinates, which is mm. Additionally, the area is considered occluded if the following three conditions are met: First, the ray did indeed hit a triangular facet in the scene, meaning the hit result was not null. Secondly, the piece of dough that was hit was not the dough itself; Third, the distance from the ray's hit point to the starting point is less than the distance from the center of the patch to the camera's optical center, allowing for a 2-unit tolerance, indicating that the hit point lies on the path between the patch and the camera. Patches determined to be occluded have their weights set to zero in subsequent steps and do not participate in texture contribution.

[0063] The square of the truncated viewpoint cosine is taken as the texture blending weight of the camera for that triangular facet. The square cosine has stronger visual discrimination than the first cosine, and a smoother transition than the fourth cosine, effectively suppressing seams at the boundaries of multi-camera fusion. The weight calculation rule is as follows: ; .

[0064] The texture blending weights of each RGB camera corresponding to each triangular facet in the 3D facet mesh model are calculated sequentially using the above method. All weights are in the range of [0,1]. The weight information of all faces is integrated into a weight map, which is then output as the weight basis for subsequent texture sampling and fusion.

[0065] Step 6, Texture Space Rasterization and Multi-Camera Back-Projection Sampling. Texture space rasterization and multi-camera back-projection sampling are key steps in mapping linearized image data from multiple cameras to a 2D UV texture space. This involves combining the 3D face triangular mesh model from Step 2, the UV coordinate pairs from Step 3, the multi-camera linear image group from Step 4, and the camera-in / outside data from Step 1 with the weight map in Step 5. Subsequently, texture space bounding box rasterization, pixel-by-pixel centroid coordinate calculation, and multi-camera back-projection color sampling with validity filtering are performed sequentially, ultimately outputting independent texture buffers and weight maps for each camera. This provides the raw texture and weight data for subsequent frequency-band texture fusion.

[0066] 6.1 Texture Space Bounding Box Rasterization. For each triangular facet in the 3D mesh, obtain the 2D texture coordinates (UV coordinates) of its three vertices in the texture map. Determine the axis-aligned bounding box of the facet in texture space using the minimum and maximum values ​​of the texture coordinates of the three vertices, and convert it into an integer coordinate range in texture pixels. Simultaneously, clip it to the boundary of the texture image, which serves as the rasterization region of the facet. Degenerate faces (with empty bounding boxes or areas approaching zero) are skipped to ensure computational stability. Complete the texture space bounding box rasterization of all triangular faces using the above method, and determine the effective rasterization region of each facet.

[0067] 6.2, Calculation of Centroid Coordinates Per Pixel. For each texture pixel within the bounding box, using the normalized texture coordinates of its pixel center as the target point, the centroid weight of the texture pixel relative to the three vertices of the triangular face is calculated using the centroid coordinate formula. Let the texture coordinates of the three vertices of the face be... The normalized coordinates of the texture pixel center are Then first calculate the denominator. : .

[0068] Then, the three centroid coordinate components are calculated sequentially. : ; ; .

[0069] Only when , and When both conditions are met, the texture pixel is determined to be located inside the patch and is a valid pixel, participating in subsequent back-projection color sampling; otherwise, the pixel is skipped. The above determination also precisely limits the rasterization range to the triangular area to avoid incorrect filling of bounding box corner pixels.

[0070] 6.3 Multi-camera back-projection color sampling. For each valid texture pixel located inside the patch, its barycentric coordinates, calculated in step 6.2, are used to interpolate the pre-projected pixel coordinates of the three vertices of the patch in each camera image to obtain the back-projected coordinates of the texture pixel in the corresponding camera image. Let the vertices be... In the The coordinates of the projected pixels in the road camera image are Then the sampling coordinates of the texture pixel in the camera image are: .

[0071] Before color sampling, a double validity filter is performed: First, it is verified that the depth values ​​of the three vertices of the face in the camera coordinate system are all positive, excluding faces that are not visible behind the camera. Second, it is verified that the back-projection coordinates are within the effective size range of the camera image, excluding pixels projected outside the image. After the above filtering, color values ​​are directly sampled from the linearized image of the camera using integer pixel coordinates. The sampled colors and their corresponding viewpoint cosine weights are written into the camera's independent texture buffer and weight map, respectively. The back-projection color sampling of all valid texture pixels in each camera is completed sequentially using the above method, and finally, the independent texture buffer and weight map of each RGB camera are output.

[0072] Step 7: Frequency-segmented multi-camera texture fusion based on Laplacian and Gaussian pyramids. A frequency-segmented fusion strategy is adopted to differentially fuse high-frequency details and low-frequency tones of the texture. This involves sequentially constructing Laplacian and Gaussian pyramids, differential fusion of optimal camera-exclusive high-frequency layer and weighted average low-frequency layer, pyramid reconstruction, and finally outputting the fused linear texture image.

[0073] 7.1 Constructing Laplacian and Gaussian Pyramids. For the texture images in the texture buffers of each camera obtained in step 6, a Laplacian pyramid is constructed by repeatedly performing downsampling and upsampling, followed by layer-by-layer subtraction. Downsampling uses Gaussian filtering combined with interlaced sampling, while upsampling uses bilinear interpolation. Each intermediate layer of the Laplacian pyramid stores the high-frequency detail residuals at the corresponding scale, and the top layer stores the lowest-frequency base tone image. Then, the normalized weight map of each camera obtained in step 6 is simultaneously subjected to repeated downsampling operations. This constructs a Gaussian weight pyramid with the same number of layers as the Laplacian pyramid of the texture image. Furthermore, the downsampling of the weight map also uses Gaussian filtering combined with interlaced sampling to achieve spatial smoothness of the weights, avoiding abrupt weight changes at the fusion boundary and providing a smooth weight basis for subsequent differential fusion.

[0074] 7.2 High-frequency layer competitive selection and low-frequency layer weighted averaging. For each high-frequency layer of the Laplacian pyramid (all intermediate layers except the top layer), the camera number corresponding to the maximum weight of each pixel is selected from the Gaussian weight maps of the corresponding layers of each camera. This camera exclusively enjoys the high-frequency detail contribution of that pixel. The high-frequency contribution of other cameras at that pixel is set to zero. This optimal camera exclusivity strategy can avoid texture blurring caused by the overlapping of high-frequency details from multiple cameras and preserve the sharp details of the optimal viewpoint to the maximum extent. For the low-frequency tonal image of the top layer of the Laplacian pyramid, a competitive selection strategy is not adopted. Instead, the low-frequency layers of all cameras are weighted and summed with their respective normalized weights to achieve a smooth tonal transition and eliminate color block seams caused by exposure differences or white balance deviations between different cameras.

[0075] The fused low-frequency layer The calculation is as follows: in: Let represent the normalized fusion weight of the i-th camera in this layer, and satisfy the normalization constraint condition. , This represents the low-frequency tone image of the top layer of the Laplacian pyramid corresponding to the i-th camera.

[0076] 7.3 Pyramid Reconstruction. The high-frequency residual layer and the fused low-frequency layer obtained by layer-by-layer fusion according to the above strategy are combined to form a fused Laplacian pyramid. Starting from the top fused low-frequency image, upsampling (bilinear interpolation) is performed on each layer sequentially, and a high-frequency residual layer of the corresponding scale is superimposed. Layer by layer, the original texture resolution is restored to obtain a preliminary fused linear texture image. The pixel values ​​of the preliminary fused image are truncated to constrain all pixel values ​​to the floating-point range of 0-1 to prevent pixel value overflow caused by superposition operations, ensuring the physical authenticity of texture color. Finally, the fused linear texture image is output.

[0077] Step 8, Post-processing of texture. Post-processing takes the fused linear texture image obtained in Step 7 as input and sequentially performs sRGB linear space transformation and fills internal texture holes based on the Navier-Stokes equations, eliminating texture format deviations and hole defects. The final output is a detailed, seamless, and hole-free high-fidelity 3D face texture map. This texture map can be directly applied to the 3D face triangular mesh model obtained in Step 2 to form a 3D face model with high-definition texture.

[0078] 8.1, sRGB linear space conversion.

[0079] The fused texture image obtained in step 7 is represented in linear lighting space. However, both image display and texture storage use sRGB encoding space. Therefore, the linear texture image needs to be converted to sRGB encoding space and the texture direction needs to be corrected.

[0080] Let the linear brightness value be The corresponding sRGB encoded floating-point value is The piecewise function and the gamma-ray detransformation in step 4.1 are inverse operations of each other, and the transformation formula is as follows: .

[0081] The meanings of each symbol and its value are as follows: The input linear brightness value, ranging from [0,1], is obtained by fusion in step 7; : Output sRGB encoded floating-point value, with a value range of [0,1]; 0.0031308: The segmentation threshold specified by the sRGB standard (IEC61966-2-1) is the unique dividing point between linear and nonlinear segments. It is determined by standard mathematical derivation and there are no other possible values. 12.92: Linear segment proportionality coefficient, ensuring the continuous connection of two function segments at the segmentation point; 1 / 2.4≈0.4167: The exponent of the nonlinear segment power function is the reciprocal of the exponent 2.4 in step 4.1, and they are inverse operations; 1.055, 0.055: Nonlinear segment gain and offset coefficients, ensuring a continuous and smooth transition of the function at the segmentation point.

[0082] Subsequently, the sRGB floating-point value is quantized into an 8-bit unsigned integer format. The quantization formula is as follows: .

[0083] in, The value is the quantized 8-bit integer pixel value (0~255). ⌊⌋ indicates rounding down, which is the inverse operation of the normalization operation in step 4.1.

[0084] Finally, since the UV coordinate system of the 3D face mesh is flipped in the vertical direction with the image storage coordinate system, the converted sRGB texture image and mesh overlay mask are flipped along the vertical axis to ensure that the texture bonding direction is consistent with the UV unfolding direction of the model, thus ensuring the correctness of texture bonding.

[0085] 8.2 Filling Texture Holes. During texture spatial rasterization and back-projection sampling, due to factors such as camera viewpoint occlusion, UV unwrapping gaps, or projection accuracy limitations, some texture pixels within the mesh coverage area may not be effectively sampled by any camera, forming texture holes. These pixels have a weight value of 0 in the weight map, severely affecting the integrity and visual continuity of the texture. The repair process adopted in this invention is as follows: First, the weight maps of each camera obtained in step 6 are summed pixel by pixel to obtain a total weight map. The pixel value of the total weight map represents the degree to which the camera effectively sampled at that location. The pixel region with a weight value of zero within the mesh coverage mask is extracted and marked as a hole mask to clarify the location and range of the texture hole. Then, using the hole mask as a reference, an image inpainting algorithm based on the Navier-Stokes equation is used to fill the texture hole. By solving the Navier-Stokes equation, the surrounding texture information diffuses inward to fill the hole region. During the filling process, the algorithm maintains the edge continuity and color consistency of the texture, so that the filled area blends naturally with the surrounding texture without obvious repair traces. After filling all the void areas in the texture, a slight smoothing process is performed on the entire texture image to eliminate the slight color differences between the filled area and the surrounding area, restore the overall visual continuity of the texture, and finally output a detailed, seamless, and void-free 3D face high-fidelity texture map.

[0086] Furthermore, the orientations or positional relationships described in this invention are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the purpose of facilitating the description of this invention and simplifying the description, and are not intended to indicate or imply that the device or structure referred to must have a specific orientation, or to operate in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating high-fidelity texture of three-dimensional face based on multi-view high-definition images, characterized in that The method comprises the following steps: Step 1, performing multi-camera pose calibration, sequentially completing single-camera intrinsic parameter calibration, two-by-two camera extrinsic parameter calibration, and outputting camera intrinsic parameter matrix, distortion coefficient and extrinsic parameter transformation matrix between cameras; Step 2, performing face mesh model reconstruction based on depth data, taking depth data synchronously collected by a plurality of structured light cameras as input, combining the extrinsic parameter transformation matrix of step 1 to complete point cloud coarse registration and fusion, and outputting a topologically complete three-dimensional face triangular mesh model through mesh reconstruction and optimization; Step 3, performing cylindrical UV texture unfolding on the three-dimensional face triangular mesh model of step 2, mapping three-dimensional coordinates of model vertices to two-dimensional UV texture space, and outputting UV coordinate pairs of each vertex; Step 4, performing preprocessing on high-definition images collected by the plurality of cameras, sequentially completing sRGB image linearization and cross-camera brightness adjustment, and outputting a plurality of camera linear image groups with consistent brightness; Step 5, calculating camera fusion weights based on cosine square of view angles, combining the camera intrinsic parameter calibration and extrinsic parameter calibration of step 1 and the three-dimensional face triangular mesh model of step 2, and calculating texture mixing weights of each camera on mesh triangular facets; Step 6, performing texture space rasterization and multi-camera back projection sampling, sequentially completing bounding box rasterization in texture space, pixel-by-pixel barycentric coordinate calculation and multi-camera back projection color sampling, and outputting independent texture buffers and weight maps of each camera; Step 7, performing frequency band multi-camera texture fusion based on Laplacian and Gaussian pyramids, sequentially completing Laplacian and Gaussian pyramid construction, differential fusion and pyramid reconstruction, and outputting a fused linear texture image; Step 8, performing post-processing on the fused linear texture image of step 7, sequentially completing sRGB linear space conversion and texture internal hole filling based on Navier-Stokes equation, and outputting a three-dimensional face high-fidelity PBR map with rich details, no seams and no holes.

2. The method of claim 1, wherein the method further comprises: The single-camera intrinsic parameter calibration in step 1 adopts Zhang Zhengyou plane target calibration method, places a checkerboard calibration board in the camera field of view, collects a plurality of calibration images, extracts pixel coordinates of checkerboard corner points in each image, establishes a corresponding relationship between the pixel coordinates and three-dimensional coordinates in the physical coordinate system of the calibration board, calculates camera intrinsic parameter initial values based on homography matrix decomposition, introduces a distortion model constructed by radial and tangential directions, and jointly optimizes the intrinsic parameters and distortion coefficients to minimize the re-projection error in the maximum likelihood estimation framework through a nonlinear minimization algorithm, and outputs the camera intrinsic parameter matrix and the distortion coefficient. 3.The method of claim 1, wherein: In the step 1, the camera extrinsic parameter calibration step is to take the physical corner point coordinates of the checkerboard calibration board as the three-dimensional reference points of the world coordinate system, place the calibration board in the common field of view of the two cameras, collect a plurality of synchronous images, extract the two-dimensional pixel coordinates of the checkerboard corner points in each image, and establish the point correspondence between the two-dimensional pixel coordinates and the three-dimensional coordinates in the world coordinate system; based on the known camera intrinsic matrix, the camera pose is estimated by a PnP solver, and the re-projection error of the three-dimensional points on the image plane is minimized to optimize the rotation matrix and the translation vector; the solution is weighted and averaged according to the re-projection error, and the images with errors exceeding the threshold are removed, and then the relative rotation matrix and the translation vector between the two cameras are re-estimated, and the extrinsic transformation matrix between the cameras is constructed.

4. The method of claim 1, wherein the method further comprises: In the step 2, the face mesh model reconstruction is specifically divided into the following steps: filtering the depth range of the original point cloud output by each camera, applying a rigid transformation to the multi-channel point cloud using the extrinsic transformation matrix of step 1 and unifying it to the center camera coordinate system to complete coarse registration, performing fine registration and merging the multi-channel point cloud into a unified point cloud by the ICP iterative closest point algorithm, estimating the normal of each point after filtering, smoothing and surface projection of the fused point cloud, and generating a closed triangular mesh by the Poisson surface reconstruction algorithm, and outputting a topologically complete three-dimensional face triangular mesh model after mesh optimization.

5. The method of claim 1, wherein the method further comprises: In the step 3, the cylindrical UV texture unwrapping is specifically divided into the following steps: establishing a coordinate system with X axis as horizontal direction, Y axis as vertical direction and Z axis as nose tip direction, calculating the azimuth angle φ = arctan2 (X, Z) of each vertex of the three-dimensional face triangular mesh model on the XZ plane to determine the horizontal UV coordinate u, determining the vertical UV coordinate v according to the normalized position of the vertex Y value relative to the longitudinal range of the model, performing automatic interval stretching on the coordinate u and linear mapping to [0, 1], and outputting the UV coordinate pair of each vertex. 6.The method of claim 1, wherein: In the step 4, the high-definition image preprocessing is specifically divided into the following steps: linearizing the sRGB image, normalizing the pixel value in sRGB encoding format to the floating-point range of 0-1; according to the sRGB standard de-gamma transformation rule, the pixel points with normalized pixel values less than the preset low value threshold are converted by linear proportional relationship, and the pixel points exceeding the preset low value threshold are converted by power function nonlinear transformation, to obtain linear luminance value proportional to physical illumination; The cross-camera brightness adjustment is to extract the center region of each camera image as a statistical region, perform effectiveness filtering on the pixels in the statistical region and exclude dark pixels, and calculate the brightness mean value of each color channel; taking the arithmetic mean of the brightness mean values of all cameras as the global target brightness, calculating the brightness adjustment gain of each camera image by channel, performing amplitude limiting processing on the gain, multiplying the limited adjustment coefficient to the corresponding camera linear image pixel by pixel, and performing truncation processing to obtain a plurality of camera linear image groups with consistent brightness. 7.The method of claim 1, wherein: In step 5, the camera fusion weight calculation based on the cosine of the viewing angle is specifically divided into the following steps: the three-dimensional grid vertex is transformed from the world coordinate system to the camera coordinate system by using the camera extrinsic matrix, projected to the pixel plane by the intrinsic matrix, the center coordinates of each triangular patch in the three-dimensional grid of the three-dimensional face triangular mesh model are calculated, the unit view direction vector of the camera optical center and the patch center is calculated, the dot product of the unit normal vector and the unit view direction vector is calculated to obtain the cosine value, and the negative value is truncated to zero, and the square of the truncated cosine value is taken as the texture mixing weight of the camera for the patch. 8.The method of claim 1, wherein: In step 6, the texture space rasterization and multi-camera back projection sampling are specifically divided into the following steps: texture space bounding box rasterization, obtaining the UV coordinates of the three vertices of the triangular patch of the three-dimensional face triangular mesh model, determining the axis-aligned bounding box of the patch in the texture space and converting it to the pixel integer coordinate range, clipping to the texture image boundary as the rasterization area, and skipping the degenerate patch; Pixel-by-pixel barycentric coordinate calculation: for each texture pixel in the bounding box, the normalized texture coordinates of the pixel center are taken as the target point, the barycentric weights are calculated by the barycentric coordinate formula, and only when all barycentric weights are greater than or equal to 0, it is determined as an internal valid pixel of the patch; Multi-camera back projection color sampling: for the valid pixel, the back projection coordinates in the camera image are obtained by barycentric coordinate interpolation, the effectiveness filter is executed, the color is sampled from the linearized image, and the viewing angle cosine weight is written into the weight map, and the independent texture buffer and weight map of each camera are output. 9.The method of claim 1, wherein: In step 7, the multi-camera texture fusion in different frequency bands is specifically divided into the following steps: for the texture image of each camera, a Laplacian pyramid is constructed by repeatedly downsampling and upsampling subtraction, and a Gaussian weight pyramid is constructed by repeatedly downsampling the normalized weight map of each camera; For each high-frequency layer of the Laplacian pyramid, the camera corresponding to the maximum weight value is taken pixel by pixel, and the high-frequency detail contribution of the camera to the pixel is exclusive, and the high-frequency contribution of the remaining cameras is set to zero; For the low-frequency tone image of the top layer of the Laplacian pyramid, a weighted fusion strategy based on normalized weight is adopted for processing, the layered fusion result is combined into a fused Laplacian pyramid, starting from the top layer low-frequency image, layer by layer upsampling and superimposing the corresponding high-frequency residual layer, restoring to the original texture resolution and truncating the pixel value to the legal range, and outputting the fused linear texture image. 10.The method of claim 1, wherein: In step 8, the texture post-processing is specifically divided into the following steps: sRGB linear space conversion, using a piecewise function according to the sRGB standard specification to convert the linear texture image to the sRGB encoding space, and quantizing the conversion result to 8-bit unsigned integer format; Texture internal hole filling: the total weight map is obtained by pixel-by-pixel summation of the weight maps of each camera, and the pixel region with zero weight in the grid coverage mask is extracted as the hole mask; An image inpainting algorithm based on Navier-Stokes equation is used to diffuse and fill the hole region from the known texture pixels around the hole, restore the visual continuity of the texture, and output a high-fidelity realistic texture map of the three-dimensional face.

Images