Monocular image three-dimensional human body reconstruction method based on dense normal alignment

Abstract

The invention discloses a monocular image three-dimensional human body reconstruction method based on dense normal alignment, and the method comprises the steps: firstly cutting an input original RGB image, extracting a two-dimensional key point, a surface normal graph and a continuous surface embedded vector graph, and obtaining an initial human body posture shape parameter and a translation parameter of a monocular camera through a regression device; further providing a pixel-level normal alignment algorithm, establishing a corresponding relation between image pixels and human body grid vertexes by using a continuous surface embedded vector diagram, and generating an alignment normal graph; then iterative optimization is carried out, a total energy function is calculated by combining the two-dimensional key points, the surface normal graph, the continuous surface embedded vector graph and the alignment normal graph, and back propagation is carried out to update the human body posture shape parameters until convergence; and finally outputting a three-dimensional human body grid and completing reconstruction. According to the method, dense three-dimensional geometric information in the surface normal graph is fully utilized, and the accuracy and robustness of human body reconstruction in a monocular image with depth fuzziness and a special body shape are remarkably improved.

Description

Technical Field

[0001] This invention relates to the technical field of three-dimensional human body reconstruction, and in particular to a method for three-dimensional human body reconstruction based on dense normal alignment of monocular images. Background Technology

[0002] With the rapid development of virtual reality, intelligent surveillance, and digital humans, the demand for accurately reconstructing 3D human shape and pose from a single RGB image is increasing. Existing methods are mainly divided into two categories: optimization-based methods and regression-based methods. The former, including methods like SMPLify, achieves 3D human reconstruction by fitting SMPL to 2D joints. However, relying on sparse 2D cues, the reconstruction results suffer from local pose errors due to depth ambiguity and a tendency towards uniform body shape. The latter, such as HMR and CLIFF, directly regress SMPL parameters using deep learning, which is faster but relies on a large amount of high-quality labeled data. Currently, there is a lack of real-world labeled data, making it heavily reliant on pseudo-ground value labeled data obtained through optimization-based methods. The accuracy of these pseudo-ground value labeled data is not high enough, causing existing regression methods to also suffer from local pose errors due to depth ambiguity. In recent years, although some studies have introduced contour and depth information, these have mostly remained at the level of 2D or coarse-level constraints, making it difficult to handle the ambiguity from 2D to 3D in monocular images, resulting in unreliable reconstructed human pose and shape. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for 3D human reconstruction based on dense normal alignment of monocular images. This method can effectively handle the depth ambiguity problem and the problem of body shape tending to be uniform in the existing monocular human reconstruction, and effectively improve the accuracy of 3D human posture and shape reconstruction.

[0004] To achieve the above objectives, the technical solution provided by this invention is: a method for three-dimensional human reconstruction from monocular images based on dense normal alignment, comprising the following steps:

[0005] S1: Input an original RGB image containing a person taken with a monocular camera. After human target detection and image cropping, a cropped image of a fixed size is obtained.

[0006] S2: Extract features from the cropped image to obtain two-dimensional key points, surface normal maps, and continuous surface embedding vector maps; use a human parametric model SMPL and a human mesh regressor of a monocular camera to predict the cropped image, obtain the initial pose and shape parameters of the SMPL, and output the initial translation parameters of the monocular camera.

[0007] S3: Using the extracted two-dimensional key points, construct and solve the least squares problem. Calculate the solution and the initial translation parameters of the monocular camera by performing a moving average to obtain the comprehensive optimal translation parameters of the monocular camera.

[0008] S4: For the given pose and shape parameters of SMPL, generate a human body mesh. Using a differentiable renderer, set the monocular camera parameters of the differentiable renderer to the optimal translation parameters of the monocular camera, render the human body mesh, and obtain the SMPL rendering normal map. Calculate the aligned normal map using a pixel-level normal alignment algorithm. The pixel-level normal alignment algorithm utilizes a continuous surface embedding vector map to determine the unique corresponding vertex of the pixel in the human body mesh for the pixel of the cropped image's human body region. Then, it uses the SMPL rendering normal map to calculate the average normal vector of the adjacent triangular facets of the vertex as the normal vector of the pixel, thus obtaining the aligned normal map.

[0009] S5: Using the attitude and shape parameters of the SMPL as the target optimization parameters, and using the initial attitude and shape parameters of the SMPL obtained in step S2 as the initial values ​​of the target optimization parameters, start iterative optimization calculation. In each iteration, use step S4 to calculate the alignment normal map corresponding to the current target optimization parameters, and combine the two-dimensional key points, surface normal map, and continuous surface embedding vector map to calculate the total energy function. Use the backpropagation algorithm to obtain the gradient of the target optimization parameters and update the target optimization parameters to minimize energy loss. When the total energy function converges or reaches the preset maximum number of iterations, output the optimized attitude and shape parameters of the SMPL.

[0010] S6: The optimized pose and shape parameters of the SMPL are used to generate an optimized human body mesh using a linear blending skinning algorithm. The optimized human body mesh is then rendered using a renderer and superimposed onto the cropped image to obtain the 3D human body reconstruction result of the monocular image.

[0011] Furthermore, in step S2, two-dimensional key points are extracted using the two-dimensional key point detector OpenPose and represented as follows: The surface normal map is extracted using the Sapiens surface normal regressor and represented as follows: The dense correspondence regressor (DensePose) is used to extract the continuous surface embedding vector map, which is then represented as follows: The This is a feature map with the same resolution as the cropped image, where each pixel p corresponds to a 16-dimensional embedding vector. , Represents the set of real numbers. This indicates that the SMPL is in a 16-dimensional real space; the initial pose parameters of the SMPL are predicted using the human mesh regressor CLIFF. and initial shape parameters CLIFF simultaneously outputs the initial translation parameters of the monocular camera, expressed as... .

[0012] Furthermore, step S3 includes the following steps:

[0013] S31: Use a complete perspective projection model for the monocular camera and set the focal length parameter of the monocular camera to... ,in and These are the height and width of the image, respectively;

[0014] S32: Given and Under the condition of, let Let the translation variable of the monocular camera to be solved be represented as the two-dimensional projection points of the 3D joints of SMPL. , The calculation formula is as follows:

[0015] ;

[0016] In the formula, Indicates a given and 3D joints of SMPL under the following conditions This represents the projection matrix of a monocular camera. Specifically, it is represented in the following form:

[0017] ;

[0018] S33: Given and Under the given conditions, solve the following least squares problem:

[0019] ;

[0020] The above equation can be solved using the singular value decomposition method, and the solution obtained is the initial optimal translation parameters of the monocular camera. ;

[0021] S34: Yes and By performing positional moving average calculations, the overall optimal translation parameters of the monocular camera are obtained. The calculation formula is as follows:

[0022] ;

[0023] In subsequent optimization, the translation parameters of the monocular camera were fixed at [value]. The focal length parameter is fixed as .

[0024] Furthermore, step S4 includes the following steps:

[0025] S41: For a given SMPL attitude parameters and shape parameters Human body meshes are generated using a linear blending skinning algorithm. ;

[0026] S42: Read a pre-built static lookup table that stores the predefined SMPL vertex-to-embedding vector mapping. ,in It is the vertex set of SMPL, using Indicates that the index is The 16-dimensional embedding vector corresponding to the vertex; defined For pixel-to-vertex mapping, for any pixel of the human body region in the cropped image... , This represents the index of the grid vertex with the smallest embedding vector distance to the pixel. The calculation formula is as follows:

[0027] ;

[0028] In the formula, This represents the total number of vertices in the SMPL.

[0029] S43: Use the differentiable renderer NMR and set the monocular camera translation parameters of NMR to... The focal length parameter is set to ,right Perform normal rendering to obtain the SMPL rendering normal map. ,use Represents pixels exist The corresponding normal vector in;

[0030] S44: Will The first in Each triangular facet is represented as ,definition for The corresponding normal vector, The specific calculation formula is as follows:

[0031] ;

[0032] In the formula, express The number of pixels covered, express The first covered 1 pixel;

[0033] S45: Definition For vertex-to-pixel mapping, for The index is vertex , express The normal vector of the corresponding pixel, The specific calculation formula is as follows:

[0034] ;

[0035] In the formula, Indicates and The set of adjacent triangular facets, Indicates and The number of adjacent triangular facets;

[0036] S46: Generate an aligned normal map with the same size as the cropped image. The alignment normal map It is based on the given and Generated, defined For pixels exist The corresponding normal vector in The calculation formula is as follows:

[0037] ;

[0038] In the formula, This represents the set of pixels representing the human body region in the cropped image. This represents the zero vector, used to set the normal vector of pixels in non-human areas to zero.

[0039] Furthermore, step S5 includes the following steps:

[0040] S51: Using PyTorch as the backpropagation framework, and employing the PyTorch form of SMPL, and As the target optimization parameters, a PyTorch version of NMR is applied, and the Adam optimizer is used to update the target optimization parameters. The preset maximum number of iterations is [value missing]. The Adam optimizer is an optimization algorithm based on adaptive learning rate in PyTorch.

[0041] S52: Constructing the total energy function It includes five error terms: two-dimensional joint reprojection error term. Dense vertex reprojection error term Dense normal alignment error term Global attitude priors and local attitude priors ;set up For error weight parameters, The specific form is as follows:

[0042] ;

[0043] To minimize the 2D joint projection error of 3D joints on the cropped image:

[0044] ;

[0045] In the formula, This represents the number of joints in SMPL. It is OpenPose for the first The confidence level estimated for each key point. It is a robust, differentiable Geman-McClure function. In the current given and The first SMPL under The three-dimensional position of each joint. The first one predicted by OpenPose The two-dimensional position of each joint;

[0046] The expression used to measure the error between the pixels of the human body region in a cropped image and the two-dimensional projection position of its corresponding SMPL surface vertices is:

[0047] ;

[0048] The error between the normal vector of a pixel in a cropped image and the normal vector of its corresponding SMPL surface vertex is used to measure the error after pixel-level normal alignment. Its expression is:

[0049] ;

[0050] Used to constrain attitude within a reasonable range, its expression is:

[0051] ;

[0052] The formula utilizes a Gaussian mixture model with eight Gaussian components trained on the CMU dataset. It is the first in this Gaussian mixture model The weights of the Gaussian components, and They are the first The mean vector and covariance matrix corresponding to each Gaussian component Indicates the first Gaussian components are input as The probability density function value at that location. The solver constant;

[0053] Used to punish unnatural bending of the elbows and knees to maintain anatomical correctness:

[0054] ;

[0055] In the formula, Representing the SMPL Rotation parameters of each joint. It is an exponential function that can impose a significantly increased penalty value on joint rotation angles that exceed the normal range of physiological activity;

[0056] S53: Perform iterative calculations, and in each iteration, for... and Regenerate the alignment normal map using the process in step S4, and recalculate... And obtain it by backpropagation using PyTorch. and The gradient is updated using the Adam optimizer. and ;exist Convergence or the number of iterations reached The iterative calculation then ends, and the optimized attitude parameters of the SMPL are output. and shape parameters .

[0057] Furthermore, in step S6, according to and An optimized human body mesh was generated using a linear blending skinning algorithm; a renderer was used, and the monocular camera translation and focal length parameters of the renderer were set as follows: and The optimized human body mesh image is rendered and overlaid on the cropped image to obtain the 3D human body reconstruction result of the monocular image.

[0058] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0059] 1. A surface normal map containing dense information is introduced as a three-dimensional geometric constraint. Compared with existing monocular image human body reconstruction methods that only rely on two-dimensional sparse information such as two-dimensional key points, it can effectively handle the inherent depth ambiguity problem in the human body reconstruction process of monocular images and significantly improve the accuracy of reconstructing human body shape and posture.

[0060] 2. A pixel-level normal alignment algorithm based on continuous surface embedded vector map is proposed, which solves the problem of pixel mismatch between the SMPL rendering normal map and the surface normal map, and effectively improves the accuracy and effectiveness of normal constraint.

[0061] 3. By combining dense normal alignment error terms and dense vertex reprojection error terms, the local and overall reconstruction accuracy of obese body types is effectively improved while ensuring 2D alignment accuracy.

[0062] 4. A more complete perspective projection model is adopted, and a more robust initialization method is used for the translation and focal length parameters of the monocular camera, which is closer to the real imaging process and improves the reconstruction accuracy of close-up figures.

[0063] 5. The method of the present invention is compatible with different feature extraction tools and has good scalability. Attached Figure Description

[0064] Figure 1 The SMPL used in this invention is through and A schematic diagram of generating a human body mesh.

[0065] Figure 2 This is a flowchart of the three-dimensional human body reconstruction process for the input original RGB image according to the present invention.

[0066] Figure 3 This is a visualization of the target value generated during the image preprocessing stage of this invention.

[0067] Figure 4 This is a flowchart illustrating the calculation process of the pixel-level normal alignment algorithm in this invention.

[0068] Figure 5 This is a schematic diagram illustrating an example of the results of three-dimensional human body reconstruction from a monocular image according to the present invention. Detailed Implementation

[0069] The present invention will be further described below with reference to specific embodiments.

[0070] This embodiment discloses a method for 3D human reconstruction based on dense normal alignment in monocular images. This method mainly utilizes information such as dense surface normals extracted from cropped images and introduces a pixel-level normal alignment algorithm to constrain the optimization process of SMPL pose and shape parameters. This addresses the depth ambiguity problem in the pose reconstruction results of human bodies from monocular images in existing methods, as well as the problem that the body shape of the reconstruction results tends to be average, effectively improving the reconstruction accuracy of 3D human pose and shape.

[0071] like Figure 1 As shown, the Human Parametric Model (SMPL) is a widely used linear model that takes 72-dimensional pose parameters as input. and 10-dimensional shape parameters To control the deformation of the human body mesh, among which It is a three-dimensional axial angle representation of each joint, used to control the rotation of the joint. It controls the body shape attributes of the human body, such as height and build. This model can be based on given... and Using a linear hybrid skinning algorithm, the corresponding human body mesh is finally output.

[0072] like Figure 2 The diagram illustrates the logical flow of the entire method, which mainly includes an image preprocessing stage, a monocular camera adjustment stage, and an SMPL parameter fitting stage. The image preprocessing stage extracts target values ​​to serve as initial values ​​and error metrics for the SMPL parameter fitting stage. The target values ​​extracted in the image preprocessing stage include 2D keypoints, surface normal maps, continuous surface embedding vector maps, initialized pose parameters, initial shape parameters, and initial translation parameters of the monocular camera. Different existing methods can be used to extract this data, as long as the data format requirements are met, i.e., the size and dimensionality of the output results are consistent.

[0073] In the image preprocessing stage, firstly, for the input raw RGB image containing a person captured by a monocular camera, the human target bounding box is obtained using the detectron2 object detection model. Then, image cropping is performed based on the human target bounding box to obtain a cropped image with a size of 256*256. For the cropped image, 25 two-dimensional keypoints are extracted using the OpenPose two-dimensional keypoint detector and represented as follows. The surface normal regressor Sapiens was used to extract the size of 256. 256 surface normal diagram, and represented as The Dense Pose algorithm is used to extract a continuous surface embedding vector map of size 256*256*16, which is then represented as... The Each pixel position p corresponds to a 16-dimensional embedding vector. ,in Represents the set of real numbers. This indicates that the object exists in a 16-dimensional real space; the initial pose parameters of the SMPL are predicted using a human mesh regressor CLIFF that correlates a human parametric model SMPL and a monocular camera. and shape parameters It also outputs the initial translation parameters of the 3D monocular camera. .

[0074] Figure 3 This section showcases some results from the image preprocessing stage. From left to right, the visualizations show the cropped image, 2D keypoints, surface normal map, and continuous surface embedding vector map. The visualization of the continuous surface embedding vector map is achieved using... The first dimension of each pixel in the image is used as the grayscale value, and a pseudo-color image is obtained by converting it using a predefined color map table in the OpenCV computer vision library.

[0075] In the monocular camera adjustment stage, unlike many existing works that use weak perspective projection approximations, this method employs a complete perspective projection model to more accurately represent the imaging geometry. The specific steps in the monocular camera adjustment stage are as follows:

[0076] 1) Set the focal length parameter of the monocular camera to [preset value]. ,in and These represent the height and width of the cropped image, respectively. Based on this, a perspective projection matrix is ​​constructed. as follows:

[0077] ;

[0078] 2) In the given and Under the condition of, let Let the translation variable of the monocular camera to be solved be represented as the two-dimensional projection points of the 3D joints of SMPL. , The calculation formula is as follows:

[0079] ;

[0080] in Indicates a given and The three-dimensional joints of SMPL.

[0081] 3) Through calculation and The difference between them will cause the monocular camera to be translated. The estimation is transformed into a least squares optimization problem:

[0082] ;

[0083] This problem can be solved using the singular value decomposition method. The solution is... These are the initial optimal translation parameters for a monocular camera.

[0084] 4) To and By performing positional moving average calculations, the overall optimal translation parameters of the monocular camera are obtained. The calculation formula is as follows:

[0085] ;

[0086] In the subsequent SMPL parameter fitting stage, the translation parameters of the monocular camera were fixed as follows: The focal length parameter is fixed as .

[0087] Figure 4 The computation flow of the pixel-level normal alignment algorithm proposed in this paper is demonstrated. For a given SMPL pose parameter... and shape parameters Human body meshes are generated using a linear blending skinning algorithm. Then, based on the continuous surface embedding vector graph, the given... and The alignment normal map is shown below. The pixel correspondence between the alignment normal map and the continuous surface embedding vector map is as follows: The specific steps for calculating the alignment normal map are as follows:

[0088] 1) Read a pre-built static lookup table that stores the predefined SMPL vertex-to-embedding vector mapping. ,in Represents the vertex set of SMPL, using Indicates that the index is The 16-dimensional embedding vector corresponding to the vertex; and Map the pixels of the cropped image and the vertices of the SMPL mesh to the same 16-dimensional space respectively; define For pixel-to-vertex mapping, for any pixel of the human body region in the cropped image... , This represents the index of the grid vertex with the smallest embedding vector distance to the pixel. The calculation formula is as follows:

[0089] ;

[0090] in This represents the total number of vertices in the SMPL.

[0091] 2) The first in Each triangular facet is represented as ,Will The set of pixels covered by the middle is represented as ,in for The number of pixels covered, express The first in 1 pixel.

[0092] 3) Use the differentiable renderer NMR and set the monocular camera translation parameters of NMR to... The focal length parameter is set to .according to Vertex position calculation The surface normal vector, and vertex position, surface normal vector and The surface topology is input into NMR for differentiable rendering to obtain the SMPL rendering normal map. ,use Represents pixels exist The corresponding normal vector. Furthermore, during the rasterization stage of differentiable rendering using NMR, each visible triangle is stored. The corresponding pixel set .

[0093] 4) Definition for The corresponding normal vector, The specific calculation formula is as follows:

[0094] ;

[0095] 5) Definition For vertex-to-pixel mapping, for The index is vertex , express The normal vector of the corresponding pixel, The specific calculation formula is as follows:

[0096] ;

[0097] In the formula, Indicates and The set of adjacent triangular facets, Indicates and The number of adjacent triangular facets.

[0098] 6) Generate an alignment normal map with the same size as the cropped image. The alignment normal map It is based on the given and Generated, defined For pixels exist The corresponding normal vector in The calculation formula is as follows:

[0099] ;

[0100] In the formula, This represents the set of pixels representing the human body region in the cropped image. This represents the zero vector, used to set the normal vector of pixels in non-human areas to zero.

[0101] In the SMPL parameter fitting stage, this method employs the PyTorch framework, which allows for backpropagation using errors. It applies a PyTorch version of NMR, using the PyTorch form of SMPL. and As optimization parameters, the Adam optimizer is used to update the optimization parameters, with a preset maximum number of iterations. The Adam optimizer mentioned above is an optimization algorithm in PyTorch based on an adaptive learning rate.

[0102] The total energy function of the fitted target in this stage It is the sum of five error terms: two-dimensional joint reprojection error term. Dense vertex reprojection error term Dense normal alignment error term Global attitude priors and local attitude priors ;set up Here are the error weight parameters, where the weights of each item are as follows: , , , , . The specific form is as follows:

[0103] ;

[0104] The calculation formulas for each item are shown below:

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] This is used to minimize the projection error of 3D joints onto the cropped image. It is a robust Geman-McClure loss function with good differentiability, which facilitates gradient backpropagation in the optimization process; In the current given and The first SMPL under The three-dimensional position of each joint point, then through... Mapped to a two-dimensional image plane; The first one predicted by OpenPose The two-dimensional position of each joint.

[0111] This measure assesses the error between the pixels of a human body region in a cropped image and the 2D projected positions of its corresponding SMPL surface vertices. It leverages the dense vertex correspondences provided by the continuous surface embedding vector map to provide a good fit to the 2D human body region contour.

[0112] It is used to measure the error between the normal vector of a pixel in the human body region of a cropped image and the normal vector error of its corresponding SMPL surface vertex. It is a pixel-level normal alignment error representation that can effectively reflect the error of local joint pose.

[0113] To constrain estimated poses that deviate from natural human pose distributions, are uncommon, or physically difficult to achieve, the value is defined as the minimum of the negative log-likelihood function. This project utilizes a Gaussian mixture model with eight Gaussian components trained on the CMU dataset. It is the first in this Gaussian mixture model The weights of the Gaussian components, and They are the first The mean vector and covariance matrix corresponding to each Gaussian component Indicates the first Gaussian components are input as The probability density function value at that location. It is a constant term introduced during the solution process for numerical stability.

[0114] Used to punish excessive bending of the elbows and knees, in order to maintain anatomical rationality. Among them, Representing the SMPL Rotation parameters of each joint. It is an exponential function that can impose a significantly increased penalty value on joint rotation angles that exceed the normal physiological range, thereby effectively guiding the optimization process to generate natural postures that conform to the laws of human kinematics.

[0115] During the SMPL parameter fitting phase, iterative calculations were performed. In each iteration, for and Regenerate the alignment normal map using the aforementioned process, and recalculate... And obtain it by backpropagation using PyTorch. and The gradient is updated using the Adam optimizer. and ;exist Convergence or the number of iterations reached The iterative calculation then ends, and the optimized attitude parameters of the SMPL are output. and shape parameters The Adam optimizer is an optimization algorithm in PyTorch based on an adaptive learning rate.

[0116] Figure 5 This displays the visualization results of 3D human body reconstruction from monocular images. From left to right, the images show the cropped image, surface normal map, 2D keypoints, continuous surface embedding vector map, optimized human body mesh projection image, and optimized human body mesh side view. The optimized human body mesh is based on... and Generated using a linear blending skinning algorithm. The optimized human body mesh projection image was generated using a renderer, with the renderer's monocular camera translation and focal length parameters set to [specific parameters]. and The optimized human body mesh image is rendered and then overlaid on the cropped image.

[0117] Will Figure 5 The image results and optimized human body mesh are saved to the hard drive to complete the 3D human body reconstruction of the monocular image.

[0118] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for 3D human reconstruction from monocular images based on dense normal alignment, characterized in that, Includes the following steps: S1: Input an original RGB image containing a person taken with a monocular camera. After human target detection and image cropping, a cropped image of a fixed size is obtained. S2: Extract features from the cropped image to obtain two-dimensional key points, surface normal maps, and continuous surface embedding vector maps; use a human parametric model SMPL and a human mesh regressor of a monocular camera to predict the cropped image, obtain the initial pose and shape parameters of the SMPL, and output the initial translation parameters of the monocular camera. S3: Using the extracted two-dimensional key points, construct and solve the least squares problem. Calculate the solution and the initial translation parameters of the monocular camera by performing a moving average to obtain the comprehensive optimal translation parameters of the monocular camera. S4: For the given pose and shape parameters of SMPL, generate a human body mesh. Using a differentiable renderer, set the monocular camera parameters of the differentiable renderer to the optimal translation parameters of the monocular camera, render the human body mesh, and obtain the SMPL rendering normal map. Calculate the aligned normal map using a pixel-level normal alignment algorithm. The pixel-level normal alignment algorithm utilizes a continuous surface embedding vector map to determine the unique corresponding vertex of the pixel in the human body mesh for the pixel of the cropped image's human body region. Then, it uses the SMPL rendering normal map to calculate the average normal vector of the adjacent triangular facets of the vertex as the normal vector of the pixel, thus obtaining the aligned normal map. S5: Using the attitude and shape parameters of the SMPL as the target optimization parameters, and using the initial attitude and shape parameters of the SMPL obtained in step S2 as the initial values ​​of the target optimization parameters, start iterative optimization calculation. In each iteration, use step S4 to calculate the alignment normal map corresponding to the current target optimization parameters, and combine the two-dimensional key points, surface normal map, and continuous surface embedding vector map to calculate the total energy function. Use the backpropagation algorithm to obtain the gradient of the target optimization parameters and update the target optimization parameters to minimize energy loss. When the total energy function converges or reaches the preset maximum number of iterations, output the optimized attitude and shape parameters of the SMPL. S6: The optimized pose and shape parameters of the SMPL are used to generate an optimized human body mesh using a linear blending skinning algorithm. The optimized human body mesh is then rendered using a renderer and superimposed onto the cropped image to obtain the 3D human body reconstruction result of the monocular image.

2. The method for 3D human reconstruction based on dense normal alignment of monocular images according to claim 1, characterized in that, In step S2, two-dimensional key points are extracted using the OpenPose two-dimensional key point detector and represented as follows: ; The surface normal map is extracted using the Sapiens surface normal regressor and represented as follows: The dense correspondence regressor (DensePose) is used to extract the continuous surface embedding vector map, which is then represented as follows: The This is a feature map with the same resolution as the cropped image, where each pixel p corresponds to a 16-dimensional embedding vector. , Represents the set of real numbers. This indicates that the SMPL is in a 16-dimensional real space; the initial pose parameters of the SMPL are predicted using the human mesh regressor CLIFF. and initial shape parameters CLIFF simultaneously outputs the initial translation parameters of the monocular camera, expressed as... .

3. The method for three-dimensional human reconstruction based on dense normal alignment of monocular images according to claim 2, characterized in that, Step S3 includes the following steps: S31: Use a complete perspective projection model for the monocular camera and set the focal length parameter of the monocular camera to... ,in and These are the height and width of the image, respectively; S32: Given and Under the condition of, let Let the translation variable of the monocular camera to be solved be represented as the two-dimensional projection points of the 3D joints of SMPL. , The calculation formula is as follows: ； In the formula, Indicates a given and 3D joints of SMPL under the following conditions This represents the projection matrix of a monocular camera. Specifically, it is represented in the following form: ； S33: Given and Under the given conditions, solve the following least squares problem: ； The above equation can be solved using the singular value decomposition method, and the solution obtained is the initial optimal translation parameters of the monocular camera. ; S34: Yes and By performing positional moving average calculations, the overall optimal translation parameters of the monocular camera are obtained. The calculation formula is as follows: ； In subsequent optimization, the translation parameters of the monocular camera were fixed at [value]. The focal length parameter is fixed as .

4. The method for three-dimensional human reconstruction based on dense normal alignment of monocular images according to claim 3, characterized in that, Step S4 includes the following steps: S41: For a given SMPL attitude parameters and shape parameters Human body meshes are generated using a linear blending skinning algorithm. ; S42: Read a pre-built static lookup table that stores the predefined SMPL vertex-to-embedding vector mapping. ,in It is the vertex set of SMPL, using Indicates that the index is The 16-dimensional embedding vector corresponding to the vertex; defined For pixel-to-vertex mapping, for any pixel of the human body region in the cropped image... , This represents the index of the grid vertex with the smallest embedding vector distance to the pixel. The calculation formula is as follows: ； In the formula, This represents the total number of vertices in the SMPL. S43: Use the differentiable renderer NMR and set the monocular camera translation parameters of NMR to... The focal length parameter is set to ,right Perform normal rendering to obtain the SMPL rendering normal map. ,use Represents pixels exist The corresponding normal vector in; S44: Will The first in Each triangular facet is represented as ,definition for The corresponding normal vector, The specific calculation formula is as follows: ； In the formula, express The number of pixels covered, express The first covered 1 pixel; S45: Definition For vertex-to-pixel mapping, for The index is vertex , express The normal vector of the corresponding pixel, The specific calculation formula is as follows: ； In the formula, Indicates and The set of adjacent triangular facets, Indicates and The number of adjacent triangular facets; S46: Generate an aligned normal map with the same size as the cropped image. The alignment normal map It is based on the given and Generated, defined For pixels exist The corresponding normal vector in The calculation formula is as follows: ； In the formula, This represents the set of pixels representing the human body region in the cropped image. This represents the zero vector, used to set the normal vector of pixels in non-human areas to zero.

5. The method for three-dimensional human reconstruction based on dense normal alignment of monocular images according to claim 4, characterized in that, Step S5 includes the following steps: S51: Using PyTorch as the backpropagation framework, and employing the PyTorch form of SMPL, and As the target optimization parameters, a PyTorch version of NMR is applied, and the Adam optimizer is used to update the target optimization parameters. The preset maximum number of iterations is [value missing]. The Adam optimizer is an optimization algorithm based on adaptive learning rate in PyTorch. S52: Constructing the total energy function It includes five error terms: two-dimensional joint reprojection error term. Dense vertex reprojection error term Dense normal alignment error term Global attitude priors and local attitude priors ;set up For error weight parameters, The specific form is as follows: ； To minimize the 2D joint projection error of 3D joints on the cropped image: ； In the formula, This represents the number of joints in SMPL. It is OpenPose for the first The confidence level estimated for each key point. It is a robust, differentiable Geman-McClure function. In the current given and The first SMPL under The three-dimensional position of each joint. The first one predicted by OpenPose The two-dimensional position of each joint; The expression used to measure the error between the pixels of the human body region in a cropped image and the two-dimensional projection position of its corresponding SMPL surface vertices is: ； The error between the normal vector of a pixel in a cropped image and the normal vector of its corresponding SMPL surface vertex is used to measure the error after pixel-level normal alignment. Its expression is: ； Used to constrain attitude within a reasonable range, its expression is: ； The formula utilizes a Gaussian mixture model with eight Gaussian components trained on the CMU dataset. It is the first in this Gaussian mixture model The weights of the Gaussian components, and They are the first The mean vector and covariance matrix corresponding to each Gaussian component Indicates the first Gaussian components are input as The probability density function value at that location. These are solver constants; Used to punish unnatural bending of the elbows and knees to maintain anatomical correctness: ； In the formula, Representing the SMPL Rotation parameters of each joint. It is an exponential function that can impose a significantly increased penalty value on joint rotation angles that exceed the normal range of physiological activity; S53: Perform iterative calculations, and in each iteration, for... and Regenerate the alignment normal map using the process in step S4, and recalculate... And obtain it by backpropagation using PyTorch. and The gradient is updated using the Adam optimizer. and ;exist Convergence or the number of iterations reached The iterative calculation then ends, and the optimized attitude parameters of the SMPL are output. and shape parameters .

6. The method for three-dimensional human reconstruction based on dense normal alignment of monocular images according to claim 5, characterized in that, In step S6, according to and An optimized human body mesh was generated using a linear blending skinning algorithm; a renderer was used, and the monocular camera translation and focal length parameters of the renderer were set as follows: and The optimized human body mesh image is rendered and overlaid on the cropped image to obtain the 3D human body reconstruction result of the monocular image.

Images