Multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm

Abstract

The application discloses a multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm, which comprises the following steps: firstly, obtaining a rough human body surface through a visual hull algorithm according to a human body mask image; secondly, using photometric consistency constraint to optimize the shape initialized from the visual hull, so as to obtain a dense human body surface model; thirdly, calculating an illumination coefficient by using the diffuse reflection principle; and finally, using a light and shade optimization algorithm to perform high-speed real-time rendering on the dense human body surface model, so as to obtain a final simulation human body model. The application can optimize the initialized rough surface by using the contrast of the gray scale image, maintaining the photometric consistency constraint and the differentiable rendering, and can effectively solve the problems of the unsmooth surface, the unobvious geometric details and the color estimation by using the diffuse reflection principle to estimate the diffuse reflectivity and the illumination. The application can optimize the human body surface by the difference of the light and shade and the color in the image.

Description

Technical Field

[0001] This invention relates to a high-precision human body reconstruction method with multiple perspectives in the field of three-dimensional human body reconstruction, specifically a multi-view human body reconstruction method based on photometric consistency matching and optimization algorithms. Background Technology

[0002] Human 3D reconstruction from multi-view images has been studied for a long time because it is crucial for many real-world applications, including motion capture, digital entertainment, and virtual try-on. However, directly estimating human geometry using only RGB images is challenging due to the high degree of shape ambiguity and complexity of the human body. Furthermore, complex clothing styles often introduce additional difficulties for human reconstruction.

[0003] To reduce the complexity of human reconstruction, statistical human models, such as SCAPE and SMPL, have been proposed to reduce the search space through parametric models constructed using principal component analysis (PCA) and hybrid skinning. 3D human reconstruction is now reformulated as a model parameter estimation problem. Despite promising results, these parametric models can only capture the shape and pose of a minimal, unclothed body. They lack the ability to represent humans with typical clothing and geometric details.

[0004] Recently, neural rendering methods have been proposed. However, these methods cannot recover accurate geometry. Due to the ambiguity between geometry and appearance, obtaining accurate shape solely through rendering loss is insufficient. Any image can be interpreted as a plane with a complex appearance or a complex geometry with a simple appearance. Deep neural networks can produce smooth surfaces because color differences between different views are overfitted by the neural network. Shallow neural networks may lead to local optima due to poor performance. Explicit addition of multi-view rendering is necessary. Figure 1 Consistency constraints are applied to ensure accurate shape recovery.

[0005] Furthermore, these neural rendering methods perform poorly at runtime. Training these methods is time-consuming because the implicit multilayer perceptron representation is not straightforward. Gradients decrease with chain rules, leading to slow convergence. Rendering time is also unacceptable because the color of each pixel needs to be inferred by the feedforward network. Increasing the render batch size to speed up rendering requires more GPU memory, and even then, it is still far from real-time rendering. Summary of the Invention

[0006] To achieve rapid, efficient, and robust reconstruction of finely detailed human body surfaces from multi-view images, addressing issues such as distortion, artifacts, and long reconstruction times, this invention proposes a multi-view human body reconstruction method based on photometric consistency matching and optimization algorithms. Specifically, it employs a photometric consistency matching-optimized surface method and a reconstruction method that utilizes subtle differences in brightness and color to recover refined geometric textures. This invention obtains a preliminary human body surface through a simple and rapid visual shell, then uses photometric consistency constraints across multiple views to reconstruct the surface, and finally utilizes a brightness and color optimization method to enhance the details of the human body's geometric texture. The method is simple and efficient, capable of recovering highly detailed 3D human body surfaces from multi-view images.

[0007] The technical solution of the present invention is as follows:

[0008] S1: Collect and obtain the original pose images of a static human body from multiple perspectives, as well as the corresponding human body mask images and camera parameters. Then, based on the original pose images and corresponding human body mask images from different perspectives, obtain the mask pose images from the corresponding perspectives.

[0009] S2: Based on the human body mask images from multiple perspectives and the corresponding camera parameters, a rough three-dimensional human body surface model is obtained in space using the visual shell algorithm.

[0010] S3: Sampling of point cloud on rough 3D human body surface model to obtain initial 3D human body surface point cloud, and generating watertight human body mesh model based on initial 3D human body surface point cloud using differentiable decomposition calculator and differentiable reconstruction.

[0011] S4: Reconstruct a dense human body surface model based on the initial 3D human body surface point cloud and watertight human body mesh model;

[0012] S5: Based on the dense human body surface model, calculate the difference between the minimum gray value and the illuminance at different viewing angles according to the principle of diffuse reflection, and obtain the illuminance coefficient of the dense human body surface model.

[0013] S6: Based on the illumination coefficient, a shading optimization algorithm is used to optimize the geometry of the dense human body surface model, generating the final simulated human body model.

[0014] In step S1, white represents the human body and black represents the background in the human body mask image, and the human body mask image is aligned with the corresponding original pose image.

[0015] In S1, the camera parameters are the camera intrinsic matrix K and the camera extrinsic rotation matrix R and translation vector t.

[0016] Specifically, S4 is:

[0017] S4.1: Calculate the contour loss based on the current 3D human body surface point cloud and human body mask images from multiple perspectives. Use the contour loss to limit the network boundary within the mask in the watertight human body mesh model, obtain the optimized 3D human body surface point cloud and update it.

[0018] S4.2: Repeat S4.1 multiple times, and take the final optimized 3D human body surface point cloud as the optimal 3D human body surface point cloud;

[0019] S4.3: Based on the optimal 3D human body surface point cloud, perform image patch matching and photometric consistency calculation on multiple masked pose images and corresponding camera parameters to obtain a 3D sparse point cloud;

[0020] S4.4: Repeat S4.3 multiple times to filter and expand the final obtained 3D sparse point cloud to obtain a 3D dense point cloud;

[0021] S4.5: Generate a dense human body surface model based on the three-dimensional dense point cloud using the Poisson surface reconstruction method.

[0022] In S4.1, the formula for calculating the contour loss is as follows:

[0023]

[0024]

[0025] in, The value represents the contour loss, ||·||² represents the L2 norm, and i represents the viewpoint index, i = 1-N, M i Represents a human body mask image. This represents the rendered mask image, ζ() is the differentiable renderer, π is the camera parameter of the current viewpoint, and V and F represent the vertices and faces in the human body surface model reconstructed from the current 3D human body surface point cloud, respectively.

[0026] Specifically, S4.3 is as follows:

[0027] S4.3.1: Use the optimal 3D human body surface point cloud reconstruction to obtain the optimal human body surface mesh model, determine the image position of each region in the optimal human body surface mesh model under camera parameters at different viewpoints, and then obtain the region-based grayscale image corresponding to the mask pose image at different viewpoints.

[0028] S4.3.2: Calculate the photometric consistency between grayscale images based on regions, and use the photometric consistency to further optimize the optimal 3D human body surface point cloud to obtain a 3D sparse point cloud;

[0029] The specific image positions of each region s in the optimal human body surface mesh model under different camera parameters from different viewpoints are as follows:

[0030] Each region s is represented by its center point and its normal. Each region s corresponds to a pixel block q on the mask pose image from different viewpoints. After rendering the pixel block q using a differentiable renderer, the 3D positions of each pixel in the pixel block q from different viewpoints in region s are obtained, thus obtaining the pixel block point cloud from different viewpoints. The specific formula is as follows:

[0031]

[0032] in, V' and F' represent the vertex and face of the human surface model reconstructed from the optimal 3D human surface point cloud, respectively; π represents the camera parameters of the current viewpoint, and π(V') represents the vertex position calculated from the current viewpoint camera parameters.

[0033] Specifically, S5 is:

[0034] By changing the illumination coefficient, the difference between grayscale values ​​and illumination at different viewing angles is calculated based on the dense human body surface model. The illumination coefficient corresponding to the smallest difference between grayscale values ​​and illumination at different viewing angles is taken as the illumination coefficient of the dense human body surface model. The specific formula is as follows:

[0035]

[0036] in, Let ||·||² represent the minimum estimated difference, ||·||² represent the L2 norm, and n represent the number of viewpoints. 2 This represents a comparison between two perspectives, where x represents spatial location and l represents the position of the viewpoint. i It is the spherical harmonic coefficient, Y i () is formed by the normal n of the model surface. x The spherical harmonic function G() is determined to convert a color image or pixel into a grayscale image or grayscale value function, and π(x) represents the spatial position based on the current viewpoint camera parameters.

[0037] Specifically, S6 is:

[0038] S6.1: Based on the illumination coefficient, extract the albedo of the human body surface from the masked pose image under multiple views;

[0039] S6.2: Construct the optimization loss function and regularization term. Based on the albedo of the human body surface and the backpropagation of the masked pose images from multiple viewpoints, optimize the geometric structure of the dense human body surface model using the optimization loss function and regularization term to generate the final simulated human body model. The formulas for the optimization loss function and regularization term during optimization are as follows:

[0040]

[0041] in, Let |·| represent the value of the optimization loss function, and |·| represent the L1 norm. I represents the albedo value mapped from spatial location x onto the interpolated albedo map. x The color representing the spatial location x mapped onto the mask pose image;

[0042]

[0043]

[0044]

[0045] in, This represents the value of the regularization function. Let L represent the first regularization function value and the second regularization function value, respectively; let L represent the Laplacian penalty function value; and let V" represent the position of each vertex on the dense human body surface model. a ' l ' bedo This represents the albedo of each vertex on a dense human body surface model.

[0046] The beneficial effects of this invention are:

[0047] By adopting the above technical solutions, the method of the present invention can quickly restore a high-detail three-dimensional human body surface through multi-view images, restore the human body surface by using photometric consistency constraints between multiple viewpoints, and obtain the texture details of the human body surface by using a shading optimization algorithm.

[0048] This invention can use a simple shading model to speed up rendering because, generally speaking, the colors of human skin and clothing mainly come from diffuse reflection. At the same time, more detailed shapes can be recovered.

[0049] The method used in this invention does not require a large amount of high-precision datasets and can directly obtain the three-dimensional human body surface through algorithms. Furthermore, benefiting from the fact that it does not require training based on existing information, the method exhibits strong robustness. In addition, the simplicity and effectiveness of the method allow for the acquisition of high-precision human body surfaces within just a few minutes, making it highly valuable for industrial applications. Attached Figure Description

[0050] Figure 1 This is an overall flowchart of a multi-view human body reconstruction method based on photometric consistency matching and optimization algorithms according to an embodiment of the present invention.

[0051] Figure 2 This is a simple framework for multi-view human body reconstruction according to an embodiment of the present invention.

[0052] Figure 3This is a visualization scheme flow and a display of gradient propagation path in an embodiment of the present invention. Detailed Implementation

[0053] To make the above-mentioned objectives, features and advantages of the present invention easier to understand, the present invention will be described in detail below with reference to the accompanying drawings of the embodiments of the present invention, and the technical solutions of the embodiments of the present invention will be clearly and completely described, but this does not constitute a limitation on the present invention.

[0054] The hardware platform in this embodiment uses an Intel i9-12900X CPU and an NVIDIA GeForce GTX3090Ti graphics card. The system program is written in Python and uses the PyTorch, NumPy, OpenCV, and nvdiffrast libraries.

[0055] The specific embodiments and implementation processes of the present invention are as follows:

[0056] like Figure 1 and Figure 3 As shown, the present invention includes the following steps:

[0057] S1: Acquire and obtain the original pose images of a static human body from multiple viewpoints, along with the corresponding human body mask images and camera parameters. Then, based on the original pose images and corresponding human body mask images from different viewpoints, obtain the mask pose image for the corresponding viewpoint. In the human body mask image M, white represents the human body, and black represents the background. The human body mask image is aligned with the corresponding original pose image. The camera parameters are the camera intrinsic matrix K and the camera extrinsic rotation matrix R and translation vector t.

[0058] In practice, a cluster of cameras, triggered simultaneously, takes photos of the target person, capturing the image at the same moment, indicating that the person's posture remains fixed. Alternatively, a single camera (phone) can be used to film a target person in a fixed posture as they walk around in a circle.

[0059] For the same human body in the same pose obtained from different perspectives, the tool is used to obtain the camera intrinsic parameters K and camera extrinsic parameters R and t from each perspective, and to eliminate the distortion of the original photo.

[0060] Generate a human body mask image from multi-view human body images, making the parts where the human appears white and the background black, thus separating the human and the background and marking the human body that needs to be reconstructed.

[0061] S2: Based on the human body mask images from multiple perspectives and the corresponding camera parameters, a rough 3D human body surface model is obtained in space using a visual shell algorithm. You can save it as an OBJ file to view the reconstruction results;

[0062] Specifically:

[0063] Using the multi-view human mask images obtained in step S1, the human mask images are projected onto spatial coordinates using camera intrinsic and extrinsic parameters. Visible parts are marked as 1, and invisible parts are marked as 0.

[0064] Finally, there are two decision-making methods for whether a pixel in the human occupancy field contains a human: one is a veto, which means that if a pixel is not visible from any viewpoint, it is considered that no human exists; the other is a majority vote, which means that if the ratio of views containing a human is higher than a certain threshold, the pixel is marked as containing a human, otherwise it is marked as not containing a human.

[0065] Then, the Marching Cubes algorithm is used to convert the human body occupancy field into the human body surface.

[0066] S3: Sampling of point cloud on rough 3D human body surface model to obtain initial 3D human body surface point cloud, and generating watertight human body mesh model based on initial 3D human body surface point cloud using differentiable decomposition calculator and differentiable reconstruction.

[0067] The differentiable computer is a DPSR, which efficiently solves the Poisson equation using spectral methods. This method can be used to bridge directed point clouds, implicit indicator functions, and meshes. It allows any shape to be represented as a lightweight directed point cloud and efficiently generates high-quality watertight meshes for these point clouds. Since both the differentiable computer and the differentiable reconstruction are differentiable, the calculated loss can be backpropagated to update the directed point cloud S. Because all computations are differentiable, gradients can be directly backpropagated to the points and normals, as shown in the following formula:

[0068] χ=DPSR(S)

[0069] Where χ represents an indicator, with 1 inside the human body and 0 outside the human body; S is a point cloud sampled from the human body surface. DPSR() represents a differentiable decomposable function.

[0070] The differentiable reconstruction method is called Differentiable Marching Cubes, and the formula is as follows:

[0071]

[0072] Where V and F represent the vertices and faces of the watertight human body mesh model, respectively. For a watertight human body mesh model, DMC() represents a differentiable reconstruction algorithm. Step S3 is globally differentiable.

[0073] S4: Reconstruct a dense human body surface model based on the initial 3D human body surface point cloud and watertight human body mesh model;

[0074] S4.1: Calculate the contour loss based on the current 3D human body surface point cloud and human body mask images from multiple perspectives. Use the contour loss to limit the network boundary within the mask in the watertight human body mesh model, obtain the optimized 3D human body surface point cloud and update it.

[0075] In S4.1, the formula for calculating contour loss is as follows:

[0076]

[0077]

[0078] in, The value represents the contour loss, ||·||² represents the L2 norm, and i represents the viewpoint index, i = 1-N, M i Represents a human body mask image. This represents the rendered mask image, ζ() is the differentiable renderer, π is the camera parameter of the current viewpoint, and V and F represent the vertices and faces in the human body surface model reconstructed from the current 3D human body surface point cloud, respectively.

[0079] S4.2: Repeat S4.1 multiple times. In practice, S4.1 is repeated 100 times, which can be adjusted according to the actual optimization situation. The final optimized 3D human body surface point cloud is taken as the optimal 3D human body surface point cloud.

[0080] S4.3: Based on the optimal 3D human body surface point cloud, perform image patch matching and photometric consistency calculation on multiple masked pose images and corresponding camera parameters to obtain a 3D sparse point cloud;

[0081] S4.3 specifically refers to:

[0082] S4.3.1: Use the optimal 3D human body surface point cloud reconstruction to obtain the optimal human body surface mesh model, determine the image position of each region in the optimal human body surface mesh model under camera parameters at different viewpoints, and then obtain the region-based grayscale image corresponding to the mask pose image at different viewpoints; the grayscale image of each region is the same in different mask pose images.

[0083] The image location of each region s in the optimal human body surface mesh model under different camera parameters from different viewpoints is as follows:

[0084] Each region s is represented by its center point and its normal. Each region s corresponds to a pixel block q on the mask pose image from different viewpoints. After rendering the pixel block q using a differentiable renderer, the precise 3D position of each pixel in the pixel block q from different viewpoints in region s is obtained, thus obtaining the pixel block point cloud from different viewpoints. The specific formula is as follows:

[0085]

[0086] in, q represents the pixel block point cloud, which is the precise 3D position of each pixel in the pixel block q in the region s. Each pixel is represented by its corresponding 3D position in camera coordinates. V' and F' represent the vertices and faces in the human surface model reconstructed from the optimal 3D human surface point cloud, respectively. π represents the camera parameters of the current viewpoint, and π(V) represents the vertex position calculated from the current viewpoint camera parameters.

[0087] The pixel block point cloud from different viewpoints is converted into the same viewpoint using the following formula, i.e., pixel block projection is performed:

[0088]

[0089] in, This represents the pixel block point cloud of region s from the source viewpoint. Represents the point cloud of all pixel blocks from the source viewpoint, π s () represents a function that transforms a point cloud in world coordinates to the source viewpoint. This represents a function that transforms the point cloud from the reference viewpoint to the world coordinate system. This represents the pixel patch point cloud obtained from region s under the reference viewpoint, where s represents a small patch of the optimal human surface mesh model, the subscript s indicates source, and the subscript r indicates reference. Indicates interpolation operation;

[0090] S4.3.2: Calculate the photometric consistency between grayscale images based on regions, and use the photometric consistency to further optimize the optimal 3D human body surface point cloud to obtain a 3D sparse point cloud;

[0091] Photometric consistency specifically involves converting a color image I to a grayscale image G, maximizing the normalized cross-correlation between the source and reference blocks. The specific formula is as follows:

[0092]

[0093] Among them, NCC(G r (s),G s (s) represents the grayscale value G of block s in the reference image. r (s) and grayscale value G in the source image s The normalized cross-correlation between (s) is represented by Cov(), where Cov() represents the covariance and Var() represents the variance.

[0094] The method compares the depth of the rendered patches with the thickness of the reprojected patches, discarding patches with significant differences. Furthermore, the method only considers blocks with NCC scores above a certain threshold to further ensure that these blocks are visible on all source views.

[0095] Furthermore, the method employs multi-view photometric consistency loss to optimize the mesh:

[0096]

[0097] The threshold δ is defined as:

[0098]

[0099] Where, δ d It is the depth threshold, δ ncc It is the NCC threshold. and These represent the rendering patch depth and the reprojection patch depth corresponding to the small region s, respectively. This represents the depth value of block s in the reference image. This represents the function that transforms the point cloud from the reference viewpoint to the world coordinate system, π. s () represents a function that transforms a point cloud in the world coordinate system to the source viewpoint.

[0100] In this step, the method obtains the human body surface optimized based on photometric consistency constraints between multiple views.

[0101] S4.4: Repeat S4.3 multiple times. In practice, S4.3 is repeated 10 times, which can be adjusted according to the actual optimization situation. This filters and expands the 3D sparse point cloud to obtain a 3D dense point cloud.

[0102] S4.5: Generate a dense human body surface model based on the three-dimensional dense point cloud using the Poisson surface reconstruction method.

[0103] S5: Based on the dense human body surface model, calculate the difference between the minimum gray value and the illuminance at different viewing angles according to the principle of diffuse reflection, and obtain the illuminance coefficient of the dense human body surface model.

[0104] S5 specifically refers to:

[0105] By changing the illumination coefficient, and using the least squares method to calculate the difference between grayscale values ​​and illumination at different viewing angles based on the dense human body surface model, the illumination coefficient corresponding to the smallest difference between grayscale values ​​and illumination at different viewing angles is taken as the illumination coefficient of the dense human body surface model. The specific formula is as follows:

[0106]

[0107] in, Let ||·||² represent the minimum estimated difference, ||·||² represent the L2 norm, and n represent the number of viewpoints. 2 This represents a comparison between two perspectives, where x represents spatial location and l represents the position of the viewpoint. i It is the spherical harmonic coefficient, Y i () is formed by the normal n of the model surface. x The spherical harmonic function G() is determined to convert a color image or pixel into a grayscale image or grayscale value function, and π(x) represents the spatial position based on the current viewpoint camera parameters.

[0108] S6: Based on the illumination coefficient, the shading optimization algorithm is used to optimize the geometry of the dense human body surface model to generate the final simulated human body model (i.e., high-precision human body model).

[0109] The main purpose of the shading optimization algorithm in S6 is to obtain the albedo of each position on the 3D human body surface and optimize the vertex positions of the human body surface model.

[0110] S6 specifically refers to:

[0111] S6.1: Based on the illumination coefficient, extract the albedo of the human body surface from the masked pose image under multiple views; the formula for the albedo map under each view is as follows:

[0112]

[0113] in, It is an interpolated albedo map, V a ' l ' bedo This represents the albedo of each vertex on a dense human body surface model.

[0114] S6.2: Construct the optimization loss function and regularization term. Based on the albedo of the human body surface and the backpropagation of the masked pose images from multiple viewpoints, optimize the geometric structure of the dense human body surface model using the optimization loss function and regularization term to generate the final simulated human body model. The formulas for the optimization loss function and regularization term during optimization are as follows:

[0115]

[0116] in, Let |·| represent the value of the optimization loss function, and |·| represent the L1 norm. I represents the albedo value mapped from spatial location x onto the interpolated albedo map. x The color representing the spatial location x mapped onto the mask pose image;

[0117] To prevent optimization from getting stuck in overfitting or local optima, a regularization term is introduced to penalize surface deformation and texture consistency.

[0118]

[0119]

[0120]

[0121] in, This represents the value of the regularization function. Let L and V represent the values ​​of the first and second regularization functions, respectively. Let L represent the Laplacian penalty function, and V" represent the position of each vertex on the dense human body surface model. a ' l ' bedo This represents the albedo of each vertex on a dense human body surface model. (Through...) Backward gradient propagation is used to obtain the albedo V at the vertices of the human body surface. a ' l ' bedo And optimize the position of the vertex V on the human body surface.

[0122] This invention proposes a multi-view human reconstruction method based on photometric consistency matching and optimization algorithms, which can be summarized into six steps as follows: Figure 2 As shown, this invention can reconstruct a static, high-detail 3D human body from multiple image perspectives. It uses photometric consistency constraints across multiple perspectives to represent the human body shape, employs a shading optimization algorithm to restore the geometric texture details of the human body surface, and utilizes differentiable rendering and a differentiable decomposition calculator to effectively solve the gradient propagation problem during the optimization process. It connects image pixels with vertices in 3D space, enabling rapid optimization and generation of high-precision 3D human bodies. Considering cost and performance, this invention is also more suitable for applying multi-view human body reconstruction methods to scenarios such as individual portrait generation, virtual humans, and metaverses.

Claims

1. A multi-view human body reconstruction method based on photometric consistency matching and optimization algorithms, characterized in that, Includes the following steps: S1: Collect and obtain the original pose images of a static human body from multiple perspectives, as well as the corresponding human body mask images and camera parameters. Then, based on the original pose images and corresponding human body mask images from different perspectives, obtain the mask pose images from the corresponding perspectives. S2: Based on the human body mask images from multiple perspectives and the corresponding camera parameters, a rough three-dimensional human body surface model is obtained in space using the visual shell algorithm. S3: Sampling of point cloud on rough 3D human body surface model to obtain initial 3D human body surface point cloud, and generating watertight human body mesh model based on initial 3D human body surface point cloud using differentiable decomposition calculator and differentiable reconstruction. S4: Reconstruct a dense human body surface model based on the initial 3D human body surface point cloud and watertight human body mesh model; S5: Based on the dense human body surface model, calculate the difference between the minimum gray value and the illuminance at different viewing angles according to the principle of diffuse reflection, and obtain the illuminance coefficient of the dense human body surface model. S6: Based on the illumination coefficient, a shading optimization algorithm is used to optimize the geometry of the dense human body surface model, generating the final simulated human body model.

2. The multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm according to claim 1, characterized in that, In step S1, white represents the human body and black represents the background in the human body mask image, and the human body mask image is aligned with the corresponding original pose image.

3. The multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm according to claim 1, characterized in that, In S1, the camera parameters are the camera intrinsic matrix K and the camera extrinsic rotation matrix R and translation vector t.

4. The multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm according to claim 1, characterized in that, Specifically, S4 is: S4.1: Calculate the contour loss based on the current 3D human body surface point cloud and human body mask images from multiple perspectives. Use the contour loss to limit the network boundary within the mask in the watertight human body mesh model, obtain the optimized 3D human body surface point cloud and update it. S4.2: Repeat S4.1 multiple times, and take the final optimized 3D human body surface point cloud as the optimal 3D human body surface point cloud; S4.3: Based on the optimal 3D human body surface point cloud, perform image patch matching and photometric consistency calculation on multiple masked pose images and corresponding camera parameters to obtain a 3D sparse point cloud; S4.4: Repeat S4.3 multiple times to filter and expand the final obtained 3D sparse point cloud to obtain a 3D dense point cloud; S4.5: Generate a dense human body surface model based on the three-dimensional dense point cloud using the Poisson surface reconstruction method.

5. The multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm according to claim 4, characterized in that, In S4.1, the formula for calculating the contour loss is as follows: in, The value represents the contour loss, ||·||² represents the L2 norm, and i represents the viewpoint index, i = 1-N, M i Represents a human body mask image. This represents the rendered mask image, ζ() is the differentiable renderer, π is the camera parameter of the current viewpoint, and V and F represent the vertices and faces in the human body surface model reconstructed from the current 3D human body surface point cloud, respectively.

6. The multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm according to claim 4, characterized in that, Specifically, S4.3 is as follows: S4.3.1: Use the optimal 3D human body surface point cloud reconstruction to obtain the optimal human body surface mesh model, determine the image position of each region in the optimal human body surface mesh model under camera parameters at different viewpoints, and then obtain the region-based grayscale image corresponding to the mask pose image at different viewpoints. S4.3.2: Calculate the photometric consistency between grayscale images based on regions, and use the photometric consistency to further optimize the optimal 3D human body surface point cloud to obtain a 3D sparse point cloud.

7. The multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm according to claim 6, characterized in that, The specific image positions of each region s in the optimal human body surface mesh model under different camera parameters from different viewpoints are as follows: Each region s is represented by its center point and its normal. Each region s corresponds to a pixel block q on the mask pose image from different viewpoints. After rendering the pixel block q using a differentiable renderer, the 3D positions of each pixel in the pixel block q from different viewpoints in region s are obtained, thus obtaining the pixel block point cloud from different viewpoints. The specific formula is as follows: in, V' and F' represent the vertex and face of the human surface model reconstructed from the optimal 3D human surface point cloud, respectively; π represents the camera parameters of the current viewpoint, and π(V') represents the vertex position calculated from the current viewpoint camera parameters.

8. The multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm according to claim 1, characterized in that, Specifically, S5 is: By changing the illumination coefficient, the difference between grayscale values ​​and illumination at different viewing angles is calculated based on the dense human body surface model. The illumination coefficient corresponding to the smallest difference between grayscale values ​​and illumination at different viewing angles is taken as the illumination coefficient of the dense human body surface model. The specific formula is as follows: in, Let ||·||² represent the minimum estimated difference, ||·||² represent the L2 norm, and n represent the number of viewpoints. 2 This represents a comparison between two perspectives, where x represents spatial location and l represents the position of the viewpoint. i It is the spherical harmonic coefficient, Y i () is formed by the normal n of the model surface. x The spherical harmonic function G() is determined to convert a color image or pixel into a grayscale image or grayscale value function, and π(x) represents the spatial position based on the current viewpoint camera parameters.

9. A multi-view human body reconstruction method based on photometric consistency matching and optimization algorithm according to claim 1, characterized in that, Specifically, S6 is: S6.1: Based on the illumination coefficient, extract the albedo of the human body surface from the masked pose image under multiple views; S6.2: Construct the optimization loss function and regularization term. Based on the albedo of the human body surface and the backpropagation of the masked pose images from multiple viewpoints, optimize the geometric structure of the dense human body surface model using the optimization loss function and regularization term to generate the final simulated human body model. The formulas for the optimization loss function and regularization term during optimization are as follows: in, The value of the optimization loss function is represented by |·|, which represents the L1 norm. I represents the albedo value mapped from spatial location x onto the interpolated albedo map. x The color representing the spatial location x mapped onto the mask pose image; in, This represents the value of the regularization function. Let L represent the first regularization function value and the second regularization function value, respectively; let L represent the Laplacian penalty function value; and let V" represent the position of each vertex on the dense human body surface model. a ' l ' bedo This represents the albedo of each vertex on a dense human body surface model.

Images