A human body three-dimensional reconstruction method based on multi-view normal enhancement and mesh carving

Abstract

The application discloses a human body three-dimensional reconstruction method based on multi-view normal enhancement and mesh carving, relates to the field of computer vision and three-dimensional reconstruction, and comprises the following steps: inputting a single color human body image, generating corresponding multi-view initial color images and basic normal maps of each view through a multi-view diffusion model, fusing the optimized normal maps by performing super-resolution processing and recombination on local image blocks and the basic normal maps of each view, constructing an initial three-dimensional mesh by using the multi-view initial color images and the basic normal maps of each view, and finally constructing a human body three-dimensional model by iteratively updating mesh vertices through gradient descent, combining with a re-meshing operation to optimize the initial three-dimensional mesh, and performing surface texture baking. The application effectively avoids the depth ambiguity problem of single-view reconstruction, effectively improves image definition, and realizes efficient fusion and accurate mapping of multi-view texture.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, specifically to a method for 3D reconstruction of the human body based on multi-view normal enhancement and mesh sculpting. Background Technology

[0002] With the rapid development of virtual reality and augmented reality technologies, the rapid reconstruction of high-fidelity 3D human body models from single color images has become a research hotspot. Existing deep learning-based methods are mainly divided into implicit representation methods and diffusion generative model methods, but they still face serious challenges in practical applications.

[0003] The first type of implicit function-based methods are limited by the ambiguity of single-image depth, often resulting in geometric "flattening" or "concavity" when dealing with self-occluded regions such as the back. In addition, the conversion of implicit representation to explicit mesh is limited by pixel resolution, making it difficult to preserve high-frequency details such as clothing wrinkles.

[0004] The second type of diffusion-based methods utilizes two-dimensional priors to optimize three-dimensional representations. While this improves texture fidelity, it still suffers from three major drawbacks: First, it is limited by the generation resolution (usually...). First, the geometric details of high curvature regions (such as knuckles and deep folds) are often blurred; second, small geometric deviations between multi-view images can lead to texture misalignment; and finally, during mesh refinement, traditional deformation algorithms are prone to topological artifacts such as mesh self-intersection or breakage due to large vertex displacements.

[0005] In the prior art, Chinese patent CN119006742 discloses a method and system for human body 3D reconstruction based on deep learning. It uses a perceptual model and occupancy network to extract features from a single image and construct initial geometry. Although it can handle non-frontal and complex background images, the method is limited by the grid cell resolution and single viewpoint sampling depth when generating geometric surfaces. It is difficult to restore subtle high-frequency geometric features such as clothing wrinkles and limb joints. Moreover, its UV mapping mechanism based on two-dimensional texture is prone to texture stretching or insufficient fidelity when dealing with complex occluded areas.

[0006] Therefore, there is an urgent need for a human body 3D reconstruction method based on multi-view normal enhancement and mesh sculpting, which can perform dual detail enhancement on the normal field, eliminate geometric artifacts, significantly improve the surface accuracy and visual fidelity of human body models in complex poses, and meet the modeling needs of high-quality 3D digital humans. Summary of the Invention

[0007] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a method for three-dimensional reconstruction of the human body based on multi-view normal enhancement and mesh carving, which can effectively solve the problems of the existing technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions; This invention discloses a method for three-dimensional reconstruction of the human body based on multi-view normal enhancement and mesh sculpting, comprising: Step S1: Input a single color human body image, and generate a multi-view initial color map and a base normal map for each view through a multi-view diffusion model; Step S2: Divide the initial color image from multiple perspectives into multiple local image blocks, perform super-resolution processing and recombination on each local image block to obtain an enhanced high-definition color image; Step S3: The enhanced high-definition color image is used to calculate the detail injection intensity matrix through a spatial-geometric attention mechanism. The enhanced high-definition color image is then spatially modulated using the detail injection intensity matrix and fused with the base normal map of each viewpoint to obtain an optimized normal map. Step S4: Construct an initial 3D mesh using the initial color image from multiple perspectives and the basic normal image from each perspective. Optimize the normal image and construct a geometric deviation energy field using the enhanced high-definition color image. Iteratively update the mesh vertices through gradient descent and combine the remeshing operation to sculpt and optimize the initial 3D mesh to obtain a refined geometric mesh. Then, use the enhanced high-definition color image to bake the surface texture of the refined geometric mesh to obtain a texture map. The texture map and the refined geometric mesh constitute a 3D human body model.

[0009] Furthermore, step S2 specifically includes: S21. Divide the initial color image from multiple perspectives into n local image blocks in the spatial resolution dimension, where the value of n ranges from 16 to 256. S22. Extract texture features from each local image patch to obtain the color feature matrix of the local image patch, and construct the corresponding feature response matrix based on the color feature matrix; S23. Vectorize the feature response matrix of each local image block to obtain the corresponding one-dimensional feature vector, and calculate the response score of each local image block based on the one-dimensional feature vector. S24. Generate texture preservation coefficients for each local image block based on response score normalization, perform super-resolution processing on each local image block, weight and reassemble the super-resolution processing results using texture preservation coefficients, and restore them according to the original spatial location to obtain an enhanced high-definition color image.

[0010] Furthermore, the construction of the corresponding feature response matrix specifically includes: For the color feature matrix of each local image patch Each element The processing yields the corresponding elements of the feature response matrix R for the local image patch. The calculation formula is: ; In the formula: Represents the color feature matrix and characteristic response matrix the number of rows, Represents the color feature matrix and characteristic response matrix The number of columns; Represents the color feature matrix No. The mean of the column; Represents the color feature matrix No. The standard deviation of the column; Represents the color feature matrix The line, number The values ​​in the column; Here is the regularization constant; The magnitude of the color gradient corresponding to the pixel. This is the adaptive gradient suppression constant.

[0011] Furthermore, step S23 specifically includes: The feature response matrix R is taken as input and transformed into a one-dimensional feature vector F of length L. The calculation formula is as follows: ; in, Represents the matrix vectorization operator; It represents the Hadamardi (or Hadama) stack; This represents the spatial location weight matrix of a Gaussian distribution. Represents the characteristic response matrix The second-order Laplace differential graph; For high-frequency component fusion coefficients; Using a one-dimensional feature vector F as input, the response score of each local image patch is calculated based on weighted fusion: ; In the formula: Represents a one-dimensional eigenvector The One element; and They represent one-dimensional eigenvectors respectively. The minimum and maximum values; Represents a one-dimensional eigenvector The mean; and These are the weighting coefficients for the linear normalization term and the exponential distribution term, respectively; In response to the sharpening index; is the regularization constant.

[0012] Furthermore, step S24 specifically includes: Local image patches are preprocessed by size normalization and edge filling, then input into a lightweight EDSR super-resolution network to extract multi-scale texture features. After upsampling using bilinear interpolation, these features are added to the principal elements of the multi-scale texture features, increasing the spatial resolution of the local image patches to four times their original value. Finally, the fused result undergoes artifact removal filtering to output high-resolution local image patches. ; Response score As input, calculate its corresponding texture preservation coefficient. and texture preservation factor Compared with high-resolution local image patches after super-resolution processing Weighted recombination is performed to generate enhanced high-definition color images. : ; ; In the formula: k′ is the traversal index of the local image patch; T is the preset response threshold; γ is the smoothing factor that controls the filtering gradient; This represents a spatial location mapping operator.

[0013] Furthermore, the calculation method for the detail injection intensity matrix in step S3 is as follows: Texture preservation coefficients corresponding to each local image patch Arranged according to the original spatial position of each image patch, and extended to the enhanced high-definition color image via a spatial broadcasting mechanism. With the same spatial resolution, the global spatial preservation coefficient matrix is ​​obtained. ; Enhanced high-definition color images and global space preservation coefficient matrix As input, the detail injection intensity matrix is ​​calculated using a spatial-geometric attention mechanism, with the following formula: ; In the formula: Injecting an intensity matrix into the details; σ represents the Sigmoid activation function; This indicates the convolution feature extraction operation; The coefficient matrix is ​​preserved for the global space; ⊙ represents the Hadamard product; β is the bias term; This represents the gradient field of a high-resolution color image.

[0014] Furthermore, in step S3, the enhanced high-definition color image is spatially modulated using a detail injection intensity matrix and then fused with the base normal map from each viewpoint. This specifically includes: Enhanced high-definition color images with detail-injected intensity matrix Λ. Spatial modulation is performed, and the high-frequency incremental components along the normal direction are calculated separately along the X, Y, and Z axes: ; The incremental components of the three channels are then concatenated to obtain the complete three-channel normal incremental data: ; In the formula: This represents the incremental data of the normal graph on the c channel, where the values ​​of c are x, y, and z, corresponding to the X, Y, and Z channels of the normal graph, respectively. These are the convolution projection weights for the corresponding channels; Indicates the convolution operation; This indicates a channel splicing operation. This refers to the complete three-channel normal increment data formed by performing X, Y, and Z channel convolution projection on the enhanced high-definition color image using the detail-injected intensity matrix, and then stitching the channel normal high-frequency increment components of each channel. Three-channel normal incremental data With the basic normal diagram By performing weighted fusion and unitization constraints, the optimized normal graph is obtained: ; In the formula: Basic normal diagram; For detail blending coefficients; Represents the L2 norm; is the numerical stability constant.

[0015] Furthermore, step S4 specifically includes: S41. Based on the 3D surface reconstruction operator, process the initial color image from multiple views and the basic normal image from each view to generate the vertex set of the initial 3D mesh. : ; In the formula: This represents the 3D surface reconstruction mapping operator; This is an initial color image from multiple perspectives; Basic normal diagram; S42. Rendering normal map based on the current mesh Rendering colors Optimize the normal graph and enhanced high-definition color images Construct a geometric deviation energy field; iteratively update the mesh vertices through gradient descent and dynamically adjust them by combining remeshing operations until convergence to obtain a refined geometric mesh; S43. Calculate the projection weights of each viewpoint onto the refined geometric mesh texture coordinates, and use the projection weights to perform weighted fusion of the multi-view enhanced high-definition color image to obtain the texture map.

[0016] Furthermore, step S42 specifically includes: Optimize the normal graph and enhanced high-definition color images As input, combined with the current mesh's rendering normal map and rendering colors Calculate the geometric deviation energy field : ; In the formula: This represents the differentiable rendering normal map of the mesh surface in the current iteration step; Represents the differentiable rendered color image of the current mesh; Indicates the current number The set of mesh vertices for the next iteration; Represents the Laplace smoothing term; and These are the weighting coefficients for the normal consistency term, the smoothing regularization term, and the photometric consistency term, respectively. Based on geometric deviation energy field The gradient descent method is used to calculate the displacement vector field of the mesh vertices, and a continuous remeshing operator is introduced. Iteratively update the mesh state until convergence yields a refined geometric mesh. The update formula for the t-th iteration is: ; ; In the formula: The learning rate for geometric deformation; This represents the gradient of the energy field relative to the current vertex; Represents a continuous regrinding operator; Indicates the first The set of mesh vertices in the next iteration, when hour, This is the initial set of 3D mesh vertices; This indicates the mesh state after a single displacement and remeshing.

[0017] Furthermore, step S43 specifically includes: Refined geometric mesh As input, the geometric normal properties of the mesh surface are extracted, and combined with the multi-view observation vectors, the projection weight matrix is ​​calculated: ; in, For refined geometric mesh The calculated visibility indicator function is 1 if and only if the mesh patch is not occluded at that viewpoint, and 0 otherwise; Based on the projection weight matrix, the enhanced high-definition color image Blended into the final texture map ; In the formula: For a single viewpoint, at the texture coordinates (u,v) of the refined geometric mesh, the texture projection weight value is obtained by multiplying the non-negative dot product of the mesh surface normal vector and the observation vector at that viewpoint with the mesh patch visibility indicator function. Represents the refined geometric mesh Extracted surface normal vector; Represents the observation vector from the corresponding viewpoint; For refined geometric mesh The calculated visibility indicator function is 1 if and only if the mesh patch is not occluded at this viewpoint, otherwise it is 0; m represents the index of the viewpoint; Represents the projection transformation function from texture space (u,v) to the m-th viewpoint image plane; This represents the enhanced high-definition color image from the m-th viewpoint; is the numerical stability constant.

[0018] Compared with known prior art, the technical solution provided by this invention has the following beneficial effects: This invention generates initial color images and basic normal maps for each viewpoint through a multi-view diffusion model. It possesses an initial multi-view representation and basic normal map with viewpoint consistency, which can stably establish accurate 3D geometric priors and effectively avoid the depth ambiguity problem of single-viewpoint reconstruction. Enhanced high-definition color images are obtained through super-resolution and recombination, which effectively improves image clarity and avoids texture blurring and misalignment. Based on the spatial-geometric attention mechanism, it completes detail optimization, accurately extracts and integrates local high-frequency increments, and significantly improves the accuracy of the normal field, making the surface geometry of the 3D model more consistent with the real human body structure. At the same time, it performs continuous remeshing sculpting based on the geometric deviation energy field, which can dynamically optimize the mesh topology and effectively eliminate geometric artifacts such as mesh self-intersection and breakage. It achieves efficient fusion and accurate mapping of multi-view textures, so that the reconstructed human 3D model still maintains high accuracy and high realism under complex postures. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 This is a flowchart illustrating a method for 3D human body reconstruction based on multi-view normal enhancement and mesh sculpting. Figure 2 This is a diagram illustrating the effect of the method in this invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] The present invention will be further described below with reference to embodiments.

[0023] Example: This embodiment presents a method for 3D human body reconstruction based on multi-view normal enhancement and mesh sculpting, such as... Figure 1 As shown, it includes: Step S1: Input a single color human body image, and generate a multi-view initial color map and a base normal map for each view through a multi-view diffusion model; Step S2: Divide the initial color image from multiple perspectives into multiple local image blocks, perform super-resolution processing and recombination on each local image block to obtain an enhanced high-definition color image; Step S2 specifically includes: S21. Divide the initial color image from multiple perspectives into n local image blocks in the spatial resolution dimension, where the value of n ranges from 16 to 256. S22. Extract texture features from each local image patch to obtain the color feature matrix of the local image patch, and construct the corresponding feature response matrix based on the color feature matrix; S23. Vectorize the feature response matrix of each local image block to obtain the corresponding one-dimensional feature vector, and calculate the response score of each local image block based on the one-dimensional feature vector. S24. Generate texture preservation coefficients for each local image block based on response score normalization, perform super-resolution processing on each local image block, weight and reassemble the super-resolution processing results using texture preservation coefficients, and restore them according to the original spatial location to obtain an enhanced high-definition color image. Constructing the corresponding feature response matrix specifically includes: For the color feature matrix of each local image patch Each element The processing yields the corresponding elements of the feature response matrix R for the local image patch. The calculation formula is: ; In the formula: Represents the color feature matrix and characteristic response matrix the number of rows, Represents the color feature matrix and characteristic response matrix The number of columns; Represents the color feature matrix No. The mean of the column; Represents the color feature matrix No. The standard deviation of the column; Represents the color feature matrix The line, number The values ​​in the column; Here is the regularization constant; The magnitude of the color gradient corresponding to the pixel. This is the adaptive gradient suppression constant; The above formula first normalizes the color feature matrix, then suppresses noise in smooth areas and highlights true boundaries through an exponential decay term, and calculates the feature response in a non-linear manner, which can accurately distinguish between effective texture and redundant noise and improve the accuracy of feature extraction. Step S23 specifically includes: The feature response matrix R is taken as input and transformed into a one-dimensional feature vector F of length L. The calculation formula is as follows: ; in, Represents the matrix vectorization operator; It represents the Hadamardi (or Hadama) stack; The spatial location weight matrix represents the Gaussian distribution and is used to evaluate the feature response matrix. Centralized weighting; Represents the characteristic response matrix The second-order Laplace derivative plot is used to extract high-frequency edge information; The high-frequency component fusion coefficient has a value range of [value range missing]. to ; The above formula fused the feature response matrix and the spatial weight matrix into a vector, and then superimposed the high-frequency edge information extracted by the second-order derivative of Laplacian. This can take into account both spatial weight and detailed features, so that the feature vector can fully carry the image texture and edge information. Using a one-dimensional feature vector F as input, the response score of each local image patch is calculated based on weighted fusion: ; In the formula: Represents a one-dimensional eigenvector The One element; and They represent one-dimensional eigenvectors respectively. The minimum and maximum values; Represents a one-dimensional eigenvector The mean; and These are the weighting coefficients for the linear normalization term and the exponential distribution term, respectively; In response to the sharpening index, a value greater than 1 is used to nonlinearly amplify the discrimination of the high-frequency response; Here is the regularization constant; This formula calculates the response score by linear normalization and exponential distribution weighting, and amplifies the high-frequency feature discrimination by using the sharpening exponent. It can accurately screen effective texture data, avoid interference from low-frequency information, and improve texture screening efficiency. Step S24 specifically includes: Local image patches are preprocessed by size normalization and edge filling, then input into a lightweight EDSR super-resolution network to extract multi-scale texture features. After upsampling using bilinear interpolation, these features are added to the principal elements of the multi-scale texture features, increasing the spatial resolution of the local image patches to four times their original value. Finally, the fused result undergoes artifact removal filtering to output high-resolution local image patches. ; Response score As input, calculate its corresponding texture preservation coefficient. and texture preservation factor Compared with high-resolution local image patches after super-resolution processing Weighted recombination is performed to generate enhanced high-definition color images. : ; in, The range of values ​​is ; This formula uses a cubic function to construct an asymmetric activation gradient and combines it with a sigmoid mapping to generate texture retention coefficients. This can amplify the difference in the retention of high and low response textures, accurately retain high-frequency effective textures, and weaken information in ineffective smoothing regions. ; In the formula: k′ is the traversal index of the local image patch. And it corresponds one-to-one with the index k in the one-dimensional feature vector F; T is the preset response threshold; γ is the smoothing factor that controls the screening gradient, and its value ranges from 5.0 to 10.0; This represents a spatial location mapping operator, used to map weighted local image patches back to the original image coordinate system to reconstruct the complete image; This formula weights the super-resolution image patch with the retention coefficient and then reassembles the image according to its original spatial position. While improving the resolution, it retains the effective texture and generates a high-definition color image with clear details and no redundant noise. It constructs an asymmetric activation gradient through a cubic function, so that the texture below the threshold and above the threshold have more extreme retention differences. The calculation method for the detail injection intensity matrix in step S3 is as follows: Texture preservation coefficients corresponding to each local image patch Arranged according to the original spatial position of each image patch, and extended to the enhanced high-definition color image via a spatial broadcasting mechanism. With the same spatial resolution, the global spatial preservation coefficient matrix is ​​obtained. ; Enhanced high-definition color images and global space preservation coefficient matrix As input, the detail injection intensity matrix is ​​calculated using a spatial-geometric attention mechanism, with the following formula: ; In the formula: An intensity matrix is ​​injected into the details to indicate the spatial locations where normal details need enhancement; σ represents the sigmoid activation function, used to normalize the attention weights to... interval; This indicates the convolution feature extraction operation; The coefficient matrix is ​​preserved for the global space; ⊙ represents the Hadamard product, i.e., element-wise multiplication; β is the bias term with a value of 0.05; The gradient field represents a high-resolution color image and is used to capture high-frequency information at the edges; The above formula integrates convolutional features, texture preservation coefficient matrix and image gradient field, and generates a detail injection intensity matrix after normalization. This matrix can accurately locate normal enhancement regions, making detail injection more targeted. Step S3: The enhanced high-definition color image is used to calculate the detail injection intensity matrix through a spatial-geometric attention mechanism. The enhanced high-definition color image is then spatially modulated using the detail injection intensity matrix and fused with the base normal map of each viewpoint to obtain the optimized normal map. Step S3 involves spatially modulating the enhanced high-definition color image using the detail injection intensity matrix and then fusing it with the base normal map from each viewpoint. Specifically, this includes: Enhanced high-definition color images with detail-injected intensity matrix Λ. Spatial modulation is performed, and the high-frequency incremental components along the normal direction are calculated separately along the X, Y, and Z axes: ; This formula combines the detail injection intensity matrix with pre-trained convolutional projection weights to transform two-dimensional color information into three-dimensional normal increments, achieving accurate mapping from photometric to geometric information, thereby providing reliable detail data for normal field optimization. The incremental components of the three channels are then concatenated to obtain the complete three-channel normal incremental data: ; This formula stitches together the three-channel normal increment components to form complete three-channel normal increment data, which fully covers the detailed changes of the normal in three-dimensional space and provides complete detailed supplements for normal field optimization. In the formula: This represents the incremental data of the normal graph on the c channel, where the values ​​of c are x, y, and z, corresponding to the X, Y, and Z channels of the normal graph, respectively. The convolutional projection weights for the corresponding channels are network parameters pre-trained using a dataset containing true labels of 3D normals, used to map 2D color gradients (photometric information) to 3D spatial normal vectors (geometric information). Indicates the convolution operation; This indicates a channel splicing operation. This represents the high-frequency normal components of each channel obtained by performing X, Y, and Z channel convolution projection on the enhanced high-definition color image using the detail-injected intensity matrix. Three-channel normal incremental data With the basic normal diagram By performing weighted fusion and unitization constraints, the optimized normal graph is obtained: ; In the formula: Basic normal diagram; This is the detail blending coefficient, used to control the injection intensity of high-frequency details; This represents the L2 norm, used to guarantee the unit length property of the normal vector; It is the numerical stability constant; The above formula combines the weighted fusion of the basic normal map and the incremental normal data and applies unit constraints, which not only preserves the consistency of the global geometric structure, but also incorporates local high-frequency details, generating an optimized normal field with higher accuracy and richer details. Step S4: Construct an initial 3D mesh using multi-view initial color images and basic normal images from each viewpoint. Optimize the normal images and construct a geometric deviation energy field using enhanced high-definition color images. Iteratively update the mesh vertices through gradient descent and combine with remeshing operations to sculpt and optimize the initial 3D mesh to obtain a refined geometric mesh. Then, use enhanced high-definition color images to bake the surface texture of the refined geometric mesh to obtain a texture map. The texture map and the refined geometric mesh constitute a 3D human body model. Step S4 specifically includes: S41. Based on the 3D surface reconstruction operator, process the initial color image from multiple views and the basic normal image from each view to generate the vertex set of the initial 3D mesh. : ; In the formula: This represents the 3D surface reconstruction mapping operator; This is an initial color image from multiple perspectives; Basic normal diagram; S42. Rendering normal map based on the current mesh Rendering colors Optimize the normal graph and enhanced high-definition color images Construct a geometric deviation energy field; iteratively update the mesh vertices through gradient descent and dynamically adjust them by combining remeshing operations until convergence to obtain a refined geometric mesh; S43. Calculate the projection weights of each viewpoint onto the refined geometric mesh texture coordinates, and use the projection weights to perform weighted fusion of the multi-view enhanced high-definition color image to obtain the texture map. Step S42 specifically includes: Optimize the normal graph and enhanced high-definition color images As input, combined with the current mesh's rendering normal map and rendering colors Calculate the geometric deviation energy field : ; In the formula: This represents the differentiable rendering normal map of the mesh surface in the current iteration step; Represents the differentiable rendered color image of the current mesh; Indicates the current number The set of mesh vertices for the next iteration; Represents the Laplace smoothing term; and These are the weighting coefficients for the normal consistency term, the smoothing regularization term, and the photometric consistency term, respectively. The above formula integrates normal consistency, mesh smoothness, and photometric consistency to construct an energy field, comprehensively constraining the mesh optimization direction, avoiding geometric deformation and distortion, and ensuring the matching degree between texture and geometry. Based on geometric deviation energy field The gradient descent method is used to calculate the displacement vector field of the mesh vertices, and a continuous remeshing operator is introduced. Iteratively update the mesh state until convergence yields a refined geometric mesh. The update formula for the t-th iteration is: ; This formula is based on the gradient of the geometric deviation energy field. It updates the mesh vertices using the gradient descent method, allowing the mesh to gradually converge towards the target geometric shape, thereby achieving precise geometric deformation adjustment and improving the mesh geometric accuracy. ; This formula dynamically optimizes the mesh topology through a continuous remeshing operator, performs edge collapse and splitting operations when the vertex is displaced, and repairs artifacts such as mesh self-intersection and fracture, making the mesh topology more regular and reasonable. In the formula: The learning rate for geometric deformation; This represents the gradient of the energy field relative to the current vertex; Represents a continuous regrinding operator; Indicates the first The set of mesh vertices in the next iteration, when hour, This is the initial set of 3D mesh vertices; This indicates the mesh state after a single displacement and remeshing; Step S43 specifically includes: Refined geometric mesh As input, the geometric normal properties of the mesh surface are extracted, and combined with the multi-view observation vectors, the projection weight matrix is ​​calculated: ; in, For refined geometric mesh The calculated visibility indicator function is 1 if and only if the mesh patch is not occluded at that viewpoint, and 0 otherwise; This formula combines the mesh surface normal, observation vector, and visibility to calculate the projection weight, and introduces a grazing angle attenuation mechanism to avoid texture distortion in edge areas and ensure the rationality and accuracy of texture projection. Based on the projection weight matrix, the enhanced high-definition color image Blended into the final texture map ; This formula generates the final texture map by weighted fusion of high-definition images based on multi-view projection weights, taking into account the effective texture information of each viewpoint, avoiding texture misalignment and stretching, and completing high-fidelity texture baking. In the formula: For a single viewpoint, at the texture coordinates (u,v) of the refined geometric mesh, the texture projection weight value is obtained by multiplying the non-negative dot product of the mesh surface normal vector and the observation vector at that viewpoint with the mesh patch visibility indicator function. Represents the refined geometric mesh Extracted surface normal vector; Represents the observation vector from the corresponding viewpoint; For refined geometric mesh The calculated visibility indicator function is 1 if and only if the mesh patch is not occluded at this viewpoint, otherwise it is 0; m represents the index of the viewpoint; Represents the projection transformation function from texture space (u,v) to the m-th viewpoint image plane; This represents the enhanced high-definition color image from the m-th viewpoint; This is a numerical stability constant used to avoid division-by-zero errors.

[0024] In this embodiment, the above method takes a single color image as input, generates a unified three-dimensional geometric prior based on multi-view diffusion, enhances high-definition texture details through nonlinear feature filtering, improves geometric accuracy by accurately optimizing the normal field, and optimizes the topology structure through continuous mesh sculpting. This effectively solves the problems of texture blurring, geometric breakage and self-intersection in single-view reconstruction, fully preserves high-frequency features, makes the surface of the human body three-dimensional model smoother and more realistic, and makes the texture baking natural and clear, thereby meeting the actual needs of high-quality digital human modeling and accurate three-dimensional human body restoration in VR / AR scenes.

[0025] See the further detailed description of the method in the above embodiments: The multi-view diffusion generation model employs a U-shaped network architecture combined with a diffusion Transformer encoder structure. The overall hierarchy covers an image encoding layer, a viewpoint alignment layer, a diffusion sampling layer, and a normal prediction layer. During training, large-scale human 3D datasets such as THuman3.0 and RenderPeople are used, with joint constraints of viewpoint consistency loss, normal reconstruction loss, and image reconstruction loss. The optimizer is AdamW, and the learning rate is set to 1 / 100,000. During the inference phase, eight viewpoints are uniformly sampled at 360 degrees to generate corresponding initial color images and basic normal images. By aligning the camera intrinsic and extrinsic parameters, a 3D human geometric prior with viewpoint consistency is constructed.

[0026] The 3D surface reconstruction mapping operator applies a Poisson reconstruction algorithm based on normal guidance. First, it extracts 3D point clouds from multi-view color images and basic normal maps, and then generates an initial 3D mesh vertex set through point cloud meshing. The continuous remeshing operator uses mesh edge length balance and normal consistency as constraints. During the iteration process, it performs edge collapse, edge splitting and vertex smoothing operations to dynamically optimize the mesh topology and avoid geometric artifacts such as mesh self-intersection and breakage during vertex displacement.

[0027] The convolutional projection weights are pre-trained using a human image dataset containing real labels for 3D normals. The dataset uses high-precision 3D scanning equipment to obtain real human normal annotation data. The training framework is built on PyTorch, and the mean square error of normal prediction is used as the training loss function. When the loss on the validation set decreases by less than 0.05% for ten consecutive rounds, the parameters are considered to have converged. Finally, the weight parameters that can accurately map the 2D color gradient to the 3D spatial normal vector are obtained.

[0028] The Gaussian distribution spatial location weight matrix uses the center of the local image patch as the mean of the Gaussian kernel, and the variance of the Gaussian kernel is set to one-quarter of the size of the local image patch. The matrix size is consistent with the resolution of the local image patch. It is generated one by one according to the spatial location of the local image patch and is used to center and weight the feature response matrix, strengthen the feature weight of the central region of the image patch, and weaken the interference information of the edge region.

[0029] The initial color image from multiple perspectives is divided into 16 to 256 local image blocks according to the rule of uniformity and non-overlap. The size of the image block is calculated and determined proportionally based on the total resolution of the image. During the segmentation process, pixel-level boundary alignment is maintained, and cross-block pixel overlap is not set. The boundary areas are filled by pixel copying to ensure the integrity of image block segmentation and subsequent reconstruction.

[0030] The super-resolution processing of local image patches uses a lightweight EDSR super-resolution model. This network contains a four-layer residual structure. The input resolution is the initial image patch size, and the output resolution is increased by four times based on the input. The inference process uses bilinear interpolation upsampling combined with convolutional feature fusion. In the post-processing stage, artifacts generated by super-resolution are removed, and finally clear high-resolution local image patches are obtained.

[0031] The visibility indicator function is calculated using an occlusion detection algorithm based on ray projection. The grid patch is projected onto the image plane of the corresponding viewpoint, and the occlusion status is determined by finding the intersection between the ray and the grid patch. The function takes a value of one when the grid patch is not occluded and a value of zero when it is occluded. The projection calculation uses a perspective projection transformation method, and the visibility determination threshold is set to a patch projection coverage of more than 95%.

[0032] The iterative convergence of mesh sculpting optimization is based on the core criterion that the change in the geometric deviation energy field is less than one ten-thousandth for twenty consecutive iterations. At the same time, a maximum number of iterations of 500 and a single-step displacement threshold of less than one millionth of the vertex are set as auxiliary termination conditions. The optimization stops when any termination condition is met during the iteration process, and the refined geometric mesh is output.

[0033] Furthermore, the method in the above embodiments applies the normal-guided Poisson reconstruction algorithm to achieve 3D surface reconstruction. First, based on the multi-view initial color image and the basic normal image, a 3D sparse point cloud is generated through SIFT feature matching and camera parameter inverse calculation. Then, the moving least squares method is applied to densify the sparse point cloud, with the point cloud density set to 2048 points per square centimeter. Subsequently, with the basic normal image as a constraint, the Poisson reconstruction depth is set to 10 levels, and the mesh sampling resolution is 2048×2048. Finally, an initial 3D mesh vertex set is generated to ensure the initial mesh... The topology is complete, without holes or missing parts. It applies a dynamic remeshing rule based on edge folding and edge splitting. The specific constraints are that the target edge length of the mesh is set to 1 / 512 of the diagonal of the model bounding box, and the normal consistency threshold is set to 15°. During the iteration process, edges with an edge length greater than 1.5 times the target edge length are split, and edges with an edge length less than 0.5 times the target edge length are folded. At the same time, Laplacian smoothing is performed on the vertices with a smoothing coefficient of 0.1 to completely avoid geometric artifacts such as mesh self-intersection, breakage, and distortion during vertex displacement.

[0034] The visibility indicator function is implemented using a ray intersection algorithm based on perspective projection. Specifically, the refined geometric mesh is projected onto the image plane of the corresponding viewpoint, and observation rays are emitted from the pixel center. When the rays intersect with the mesh, AABB bounding boxes are used to accelerate the detection. The visibility judgment threshold is set to a coverage of at least 95% for the mesh projection. When the mesh is not occluded, the rays have no intersection points and the function value is 1. When the mesh is occluded, the function value is 0, ensuring that there is no occlusion error in the texture projection.

[0035] The multi-view diffusion generation module uses the U-Net architecture with a DiffusionTransformer encoder. The image encoding layer uses 6 convolutional layers with channel numbers of 64, 128, 256, 512, 512, and 512 respectively. The viewpoint alignment layer uses a cross-viewpoint attention mechanism with 8 heads. The diffusion sampling layer has 50 sampling steps. The normal prediction layer uses 3 deconvolutional layers to output a three-channel normal map. The training dataset uses a fusion dataset of THuman 3.0 and RenderPeople, containing a total of 120,000 sets of human 3D data. The loss function uses a weighted fusion of view consistency loss, normal reconstruction loss, and image reconstruction loss, with weights of 0.4, 0.4, and 0.2, respectively. The optimizer is AdamW, with a fixed preset learning rate, a batch size of 16, and 100 training epochs. Training stops when the validation set loss decreases by no more than 5e-5 for 10 consecutive epochs. During the inference phase, 8 viewpoints are sampled uniformly, with horizontal azimuth angles of 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°, respectively, and the camera elevation angle is fixed at 10°.

[0036] The lightweight EDSR super-resolution network consists of a 4-layer residual structure. Each residual block contains two layers of 3×3 convolution and ReLU activation functions. The input image block resolution is uniformly 64×64, and the output resolution is increased by 4 times to 256×256. Bilinear interpolation is used for upsampling, and a 3×3 Gaussian filter is used for post-processing to remove super-resolution artifacts, ensuring that the super-resolution image is free of jagged edges and blur.

[0037] The convolutional projection weight network is built on the PyTorch framework. The network contains three 1×1 convolutional layers with 64, 32, and 3 channels respectively. The training dataset uses a high-precision human normal vector annotation dataset with 50,000 annotated data sets. The loss function uses the mean squared error of normal prediction. The learning rate is fixed at 5e-4, the batch size is set to 32, and the number of training epochs is 60. The network converges when the loss on the validation set decreases by no more than 5e-5 for 10 consecutive epochs. It can achieve accurate mapping from two-dimensional color gradients to three-dimensional normal vectors.

[0038] The intrinsic parameters of the cameras for the eight viewpoints are uniformly set. The horizontal parameters are 1000, 0, 256, the vertical parameters are 0, 1000, 256, and the lateral parameters are 0, 0, 1. The camera extrinsic parameters are aligned with the viewpoints through rotation and translation matrices. The rotation angle is consistent with the azimuth and elevation angles mentioned earlier. The translation vector is uniformly set to (0, 0, 2.5) to ensure that the multi-view images and normal maps have strict viewpoint consistency and no geometric deviation issues.

[0039] To avoid anomalies in normal vectors and distortion of the model, this embodiment adds normalization constraints and geometric consistency checks. After normal vector fusion, L2 norm normalization is enforced to ensure that the normal length is always 1 and there are no abnormal vectors. At the same time, angle constraints are added to the normal vectors of adjacent facets. When the angle exceeds 30°, smoothing is performed to ensure the geometric continuity of the model surface and conform to the real human body structure.

[0040] In summary, the method described in the above embodiments generates initial color images and basic normal maps for each viewpoint through a multi-view diffusion model. It possesses an initial multi-view representation and basic normal map with viewpoint consistency, which can stably establish accurate 3D geometric priors, effectively avoid the depth ambiguity problem of single-viewpoint reconstruction, and obtain enhanced high-definition color images through super-resolution and recombination, effectively improving image clarity and avoiding texture blurring and misalignment. Based on the spatial-geometric attention mechanism, it completes detail optimization, accurately extracts and integrates local high-frequency increments, significantly improves the accuracy of the normal field, and makes the surface geometry of the 3D model more closely fit the real human body structure. At the same time, it performs continuous remeshing sculpting based on the geometric deviation energy field, which can dynamically optimize the mesh topology, effectively eliminate geometric artifacts such as mesh self-intersection and breakage, and achieve efficient fusion and accurate mapping of multi-view textures, so that the reconstructed human 3D model still maintains high accuracy and high realism under complex postures.

[0041] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for three-dimensional reconstruction of the human body based on multi-view normal enhancement and mesh sculpting, characterized in that, include: Step S1: Input a single color human body image, and generate a multi-view initial color map and a base normal map for each view through a multi-view diffusion model; Step S2: Divide the initial color image from multiple perspectives into multiple local image blocks, perform super-resolution processing and recombination on each local image block to obtain an enhanced high-definition color image; Step S3: The enhanced high-definition color image is used to calculate the detail injection intensity matrix through a spatial-geometric attention mechanism. The enhanced high-definition color image is then spatially modulated using the detail injection intensity matrix and fused with the base normal maps of each viewpoint to obtain an optimized normal map. The calculation method for the detail injection intensity matrix is ​​as follows: Texture preservation coefficients corresponding to each local image patch Arranged according to the original spatial position of each image patch, and extended to the enhanced high-definition color image via a spatial broadcasting mechanism. With the same spatial resolution, the global spatial preservation coefficient matrix is ​​obtained. ; Enhanced high-definition color images and global space preservation coefficient matrix As input, the detail injection intensity matrix is ​​calculated using a spatial-geometric attention mechanism, with the following formula: ； In the formula: Injecting an intensity matrix into the details; σ represents the Sigmoid activation function; This indicates the convolution feature extraction operation; The coefficient matrix is ​​preserved for the global space; ⊙ represents the Hadamard product; β is the bias term; Represents the gradient field of a high-resolution color image; Step S4: Construct an initial 3D mesh using multi-view initial color images and basic normal maps of each viewpoint. Optimize the normal maps and construct a geometric deviation energy field using enhanced high-definition color images. Iteratively update the mesh vertices through gradient descent and combine this with remeshing operations to sculpt and optimize the initial 3D mesh, obtaining a refined geometric mesh. Then, bake the surface texture of the refined geometric mesh using enhanced high-definition color images to obtain a texture map. The texture map and the refined geometric mesh constitute a 3D human body model. Step S4 specifically includes: S41. Based on the 3D surface reconstruction operator, process the initial color image from multiple views and the basic normal image from each view to generate the vertex set of the initial 3D mesh. : ； In the formula: This represents the 3D surface reconstruction mapping operator; This is an initial color image from multiple perspectives; Basic normal diagram; S42. Rendering normal map based on the current mesh Rendering colors Optimize the normal graph and enhanced high-definition color images Construct the geometric deviation energy field; iteratively update the mesh vertices through gradient descent and dynamically adjust them using remeshization until convergence to obtain a refined geometric mesh; step S42 specifically includes: Optimize the normal graph and enhanced high-definition color images As input, combined with the current mesh's rendering normal map and rendering colors Calculate the geometric deviation energy field : ； In the formula: This represents the differentiable rendering normal map of the mesh surface in the current iteration step; Represents the differentiable rendered color image of the current mesh; Indicates the current number The set of mesh vertices for the next iteration; Represents the Laplace smoothing term; and These are the weighting coefficients for the normal consistency term, the smoothing regularization term, and the photometric consistency term, respectively. Based on geometric deviation energy field The gradient descent method is used to calculate the displacement vector field of the mesh vertices, and a continuous remeshing operator is introduced. Iteratively update the mesh state until convergence yields a refined geometric mesh. The update formula for the t-th iteration is: ； ； In the formula: The learning rate for geometric deformation; This represents the gradient of the energy field relative to the current vertex; Represents a continuous regrinding operator; Indicates the first The set of mesh vertices in the next iteration, when hour, This is the initial set of 3D mesh vertices; This indicates the mesh state after a single displacement and remeshing; S43. Calculate the projection weights of each viewpoint onto the refined geometric mesh texture coordinates, and use the projection weights to perform weighted fusion of the multi-view enhanced high-definition color image to obtain the texture map.

2. The method for three-dimensional reconstruction of the human body based on multi-view normal enhancement and mesh sculpting according to claim 1, characterized in that, Step S2 specifically includes: S21. Divide the initial color image from multiple perspectives into n local image blocks in the spatial resolution dimension, where the value of n ranges from 16 to 256. S22. Extract texture features from each local image patch to obtain the color feature matrix of the local image patch, and construct the corresponding feature response matrix based on the color feature matrix; S23. Vectorize the feature response matrix of each local image block to obtain the corresponding one-dimensional feature vector, and calculate the response score of each local image block based on the one-dimensional feature vector. S24. Generate texture preservation coefficients for each local image block based on response score normalization, perform super-resolution processing on each local image block, weight and reassemble the super-resolution processing results using texture preservation coefficients, and restore them according to the original spatial location to obtain an enhanced high-definition color image.

3. The method for three-dimensional reconstruction of the human body based on multi-view normal enhancement and mesh sculpting according to claim 2, characterized in that, The construction of the corresponding feature response matrix specifically includes: For the color feature matrix of each local image patch Each element The processing yields the corresponding elements of the feature response matrix R for the local image patch. The calculation formula is: ； In the formula: Represents the color feature matrix and characteristic response matrix the number of rows, Represents the color feature matrix and characteristic response matrix The number of columns; Represents the color feature matrix No. The mean of the column; Represents the color feature matrix No. The standard deviation of the column; Represents the color feature matrix The line, number The values ​​in the column; Here is the regularization constant; The magnitude of the color gradient corresponding to the pixel. This is the adaptive gradient suppression constant.

4. The method for three-dimensional reconstruction of the human body based on multi-view normal enhancement and mesh sculpting according to claim 2, characterized in that, Step S23 specifically includes: The feature response matrix R is taken as input and transformed into a one-dimensional feature vector F of length L. The calculation formula is as follows: ； in, Represents the matrix vectorization operator; It represents the Hadamardi (or Hadama) stack; This represents the spatial location weight matrix of a Gaussian distribution. Represents the characteristic response matrix The second-order Laplace differential graph; For high-frequency component fusion coefficients; Using a one-dimensional feature vector F as input, the response score of each local image patch is calculated based on weighted fusion: ； In the formula: Represents a one-dimensional eigenvector The One element; and They represent one-dimensional eigenvectors respectively. The minimum and maximum values; Represents a one-dimensional eigenvector The mean; and These are the weighting coefficients for the linear normalization term and the exponential distribution term, respectively; In response to the sharpening index; is the regularization constant.

5. The method for three-dimensional reconstruction of the human body based on multi-view normal enhancement and mesh sculpting according to claim 4, characterized in that, Step S24 specifically includes: Local image patches are preprocessed by size normalization and edge filling, then input into a lightweight EDSR super-resolution network to extract multi-scale texture features. After upsampling using bilinear interpolation, these features are added to the principal elements of the multi-scale texture features, increasing the spatial resolution of the local image patches to four times their original value. Finally, the fused result undergoes artifact removal filtering to output high-resolution local image patches. ; Response score As input, calculate its corresponding texture preservation coefficient. and texture preservation factor Compared with high-resolution local image patches after super-resolution processing Weighted recombination is performed to generate enhanced high-definition color images. : ； ； In the formula: k′ is the traversal index of the local image patch; T is the preset response threshold; γ is the smoothing factor that controls the filtering gradient; This represents a spatial location mapping operator.

6. The method for three-dimensional reconstruction of the human body based on multi-view normal enhancement and mesh sculpting according to claim 5, characterized in that, Step S3, which involves spatially modulating the enhanced high-definition color image using a detail injection intensity matrix and then fusing it with the base normal map from each viewpoint, specifically includes: Enhanced high-definition color images with detail-injected intensity matrix Λ. Spatial modulation is performed, and the high-frequency incremental components along the normal direction are calculated separately along the X, Y, and Z axes: ； By concatenating the incremental components of the three channels, the complete three-channel normal incremental data is obtained: ； In the formula: This represents the incremental data of the normal graph on the c channel, where the values ​​of c are x, y, and z, corresponding to the X, Y, and Z channels of the normal graph, respectively. These are the convolution projection weights for the corresponding channels; * indicates the convolution operation; This indicates a channel splicing operation. This represents the high-frequency normal components of each channel obtained by performing X, Y, and Z channel convolution projection on the enhanced high-definition color image using the detail-injected intensity matrix. Three-channel normal incremental data With the basic normal diagram By performing weighted fusion and unitization constraints, the optimized normal graph is obtained: ； In the formula: Basic normal diagram; For detail blending coefficients; Represents the L2 norm; is the numerical stability constant.

7. The method for three-dimensional reconstruction of the human body based on multi-view normal enhancement and mesh sculpting according to claim 1, characterized in that, Step S43 specifically includes: Refined geometric mesh As input, the geometric normal properties of the mesh surface are extracted, and combined with the multi-view observation vectors, the projection weight matrix is ​​calculated: ； in, For refined geometric mesh The calculated visibility indicator function is 1 if and only if the mesh patch is not occluded at that viewpoint, and 0 otherwise; Based on the projection weight matrix, the enhanced high-definition color image Blended into the final texture map ; In the formula: For a single viewpoint, at the texture coordinates (u,v) of the refined geometric mesh, the texture projection weight value is obtained by multiplying the non-negative dot product of the mesh surface normal vector and the observation vector at that viewpoint with the mesh patch visibility indicator function. Indicates from refined geometric mesh Extracted surface normal vector; This represents the observation vector from the corresponding viewpoint; m represents the index of the viewpoint. Represents the projection transformation function from texture space (u,v) to the m-th viewpoint image plane; This represents the enhanced high-definition color image from the m-th viewpoint; is the numerical stability constant.

Images