Automated Reconstruction Method, Product, Medium and Device for Human Three-Dimensional Model Based on Explicit Space

Abstract

The present invention discloses a method, product, medium and device for automatic reconstruction of a three-dimensional human body model based on explicit space, relating to the field of three-dimensional modeling. First, the present invention calibrates the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial parameters of the camera; then uses the camera to photograph the human body, and optimizes the initial parameters of the camera and performs three-dimensional reconstruction for the human body area to obtain a sparse point cloud of the human body area; reconstructs a dense point cloud of the human body object based on the sparse point cloud of the human body area; reconstructs a watertight human body surface mesh based on the dense point cloud of the human body object; optimizes the human body surface mesh to restore a three-dimensional model with more human body details; and performs visual rendering on the three-dimensional model using texture mapping to generate a three-dimensional human body model. The present invention can automatically reconstruct a three-dimensional human body model according to the explicit space of the human body, and ensure the fineness of human body details and textures.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and particularly to an automated reconstruction method, product, medium, and device for a human three-dimensional model based on an explicit space. Background Art

[0002] In the 1960s and 1970s, computer graphics began to emerge. Early research mainly focused on graphics rendering, modeling, and animation production. Later, people began to try to use computers to simulate and present three-dimensional objects. With the progress of computer graphics, three-dimensional modeling technology gradually emerged. Researchers began to try to digitize the shape, texture, and other information of actual objects and store them in the computer in the form of three-dimensional models. In the 1980s and 1990s, significant progress was made in the field of computer vision. Researchers began to focus on how to make computer systems understand and interpret images. This laid the foundation for extracting three-dimensional information from two-dimensional images and also provided technical support for the development of human three-dimensional model reconstruction technology.

[0003] With the continuous breakthrough of sensor technology, the continuous optimization of algorithms, and the development of deep learning technology in recent years, the accuracy of human three-dimensional models has been continuously improved. Currently, the application of human three-dimensional models is very extensive. In the medical field, this technology can be applied to disease diagnosis, surgical planning, medical research, etc.; in the entertainment field, this technology can be applied to animation production, character design, game modeling, etc.; in sports training, this technology can be applied to professional athlete motion analysis, etc. The development and progress of human three-dimensional model reconstruction technology will also bring more possibilities to its application fields. Therefore, higher requirements are put forward for aspects such as the accuracy of this technology.

[0004] Currently, the mainstream digital human body modeling method is based on an implicit space. For example, nerfneus is used. Its overall color of the model is good, but the geometric structure is poor, and there is also a problem of blurred edges, which cannot meet the actual needs in some application scenarios. Therefore, how to break through the current human three-dimensional model reconstruction method and explore a higher-precision modeling method and system has become an urgent problem for researchers in this field. Summary of the Invention

[0005] The object of the present invention is to provide an automated reconstruction method, product, medium, and device for a human three-dimensional model based on an explicit space, which can automatically reconstruct a human three-dimensional model according to the human explicit space and ensure the fineness of human details and textures.

[0006] To achieve the above object, the present invention provides the following solutions.

[0007] On the one hand, the present invention provides an automated reconstruction method for a human three-dimensional model based on an explicit space, including:

[0008] Calibrate the internal and external parameters of the camera according to the chessboard grid calibration method to obtain the initial parameters of the camera;

[0009] Use the camera to take pictures of the human body, and optimize the initial parameters of the camera and perform 3D reconstruction for the human body area to obtain the sparse point cloud of the human body area;

[0010] Reconstruct the dense point cloud of the human body object based on the sparse point cloud of the human body area;

[0011] Reconstruct the watertight human body surface mesh based on the dense point cloud of the human body object;

[0012] Optimize the human body surface mesh to recover a 3D model with more human body details;

[0013] Use texture mapping to perform visual rendering on the 3D model to generate a 3D human body model.

[0014] Optionally, the calibrating the internal and external parameters of the camera according to the chessboard grid calibration method to obtain the initial parameters of the camera specifically includes:

[0015] Select a chessboard grid, and use the camera to collect calibration images of the chessboard grid at different positions and angles;

[0016] Use a corner detection algorithm to find the corners of the chessboard grid in each calibration image and refine the positions of these corners through sub-pixel corner detection technology;

[0017] Specify a 3D coordinate for each corner of the chessboard grid in the world coordinate system;

[0018] Use the 2D positions of the corners in the calibration image and the corresponding 3D coordinates, and use the camera calibration algorithm to calculate the internal and external parameters of the camera;

[0019] Optimize the internal and external parameters of the camera, with the goal of minimizing the reprojection error, to obtain the optimized internal and external parameters;

[0020] Use the optimized internal and external parameters to perform distortion correction and 3D reconstruction on a new calibration image to verify the accuracy of the calibration result;

[0021] If the verification of the accuracy of the calibration result does not meet the requirements, re-perform the calibration until the accuracy of the calibration result meets the requirements, and use the optimized internal and external parameters at this time as the initial parameters of the camera.

[0022] Optionally, the using the camera to take pictures of the human body, and optimizing the initial parameters of the camera and performing 3D reconstruction for the human body area to obtain the sparse point cloud of the human body area specifically includes:

[0023] Use a camera to take pictures of a human body, extract feature points from the two-dimensional images taken each time, and perform human body region MASK segmentation to extract the feature points in the two-dimensional images and segment the human body region;

[0024] Use a feature point matching algorithm to perform feature point matching between different two-dimensional images, and use the matched feature points and the initial parameters of the camera. Use a three-dimensional reconstruction algorithm to estimate the position of each feature point in three-dimensional space to obtain a sparse point cloud of the human body region;

[0025] Run local Bundle Adjustment to optimize the initial parameters of the camera and the coordinates of the sparse point cloud of the human body region to minimize the reprojection error, and obtain the locally optimized camera parameters and sparse point cloud.

[0026] Optionally, reconstructing a dense point cloud of the human body object based on the sparse point cloud of the human body region specifically includes:

[0027] Obtain the depth information of the human body region through a multi-view stereo geometry method and the locally optimized camera parameters;

[0028] Fuse the depth information with the sparse point cloud of the human body region to generate a dense point cloud representing the human body object.

[0029] Optionally, reconstructing a watertight human body surface mesh based on the dense point cloud of the human body object specifically includes:

[0030] Extract the SMPLX model from the captured images and perform an initial match with the dense point cloud of the human body object;

[0031] Make the generated mesh coincide with the dense point cloud and maintain the geometric characteristics of the SMPLX model by globally minimizing the geometric error;

[0032] Perform line-of-sight conflict detection and resolve line-of-sight conflicts to obtain a human body model after global optimization is completed;

[0033] Refine and seal the human body model mesh to reconstruct a watertight human body surface mesh.

[0034] Optionally, optimizing the human body surface mesh to restore a three-dimensional model with more human body details specifically includes:

[0035] Optimize the human body surface mesh. On the premise of ensuring the two-dimensional to three-dimensional consistency of human body joint points and facial key points, minimize the reprojection error of the three-dimensional model on different two-dimensional images as much as possible, and at the same time perform optimization of structural priors to restore more human body details to obtain a three-dimensional model of the human body object.

[0036] Optionally, using texture mapping to perform visual rendering on the three-dimensional model to generate a three-dimensional human body model specifically includes:

[0037] Map the human body color information in the two-dimensional image to the texture map according to the corresponding relationship of solid geometry, and globally optimize the map;

[0038] Apply the optimized texture map to the three-dimensional model, and use graphics rendering technology for visual rendering to generate a three-dimensional human body model.

[0039] On the other hand, the present invention also provides a computer program product, including a computer program, which when executed by a processor implements the steps of the automatic reconstruction method of the three-dimensional human body model based on explicit space.

[0040] On the other hand, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the automatic reconstruction method of the three-dimensional human body model based on explicit space.

[0041] In yet another aspect, the present invention also provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the steps of the automatic reconstruction method of the three-dimensional human body model based on explicit space.

[0042] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0043] The present invention takes into account the human body details and texture fineness of the reconstructed human body model, and realizes the high fineness of the human body model through steps such as continuously optimizing camera parameters, reducing reprojection errors, globally minimizing geometric errors, and global mapping. Specifically, the present invention forms a set of automatic three-dimensional human body model reconstruction methods based on the explicit space of the human body by using multi-angle human body image data, and can better reduce geometric errors and optimize camera parameters during the modeling process. And the present invention optimizes and reconstructs the human body surface mesh integrating the depth information of the human body area, ensuring the surface accuracy and model compactness of the reconstructed human body, and can obtain a more refined human body reconstruction model. Brief Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0045] Figure 1 It is a flowchart of the automatic reconstruction method of the three-dimensional human body model based on explicit space provided by the present invention. Detailed Embodiments

[0046] Next, in combination with the accompanying drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0047] The object of the present invention is to provide a method, product, medium and device for automatically reconstructing a three-dimensional human body model based on an explicit space, so as to automatically reconstruct a three-dimensional human body model according to the explicit space of the human body while ensuring the fineness of human body details and textures.

[0048] To make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in combination with the accompanying drawings and specific embodiments.

[0049] See Figure 1 , a method for automatically reconstructing a three-dimensional human body model based on an explicit space provided by the present invention includes:

[0050] Step 1: Calibrate the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial parameters of the camera.

[0051] The present invention calibrates the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial values of the camera parameters. The specific steps of Step 1 include:

[0052] Step 1.1: Select a checkerboard and use the camera to collect calibration images of the checkerboard at different positions and angles.

[0053] Use a checkerboard with obvious black and white contrast as the calibration object. Place the checkerboard at different positions and angles, and use the camera to take pictures of the checkerboard to obtain sufficient image data.

[0054] Step 1.2: Use the corner detection algorithm to find the corners of the checkerboard in each calibration image and refine the positions of these corners through sub-pixel corner detection technology.

[0055] In each collected image, use the corner detection algorithm to find the inner corners of the checkerboard. Further refine the positions of these corners through sub-pixel corner detection technology to improve the accuracy of calibration.

[0056] Step 1.3: Specify a three-dimensional coordinate for each corner of the checkerboard in the world coordinate system.

[0057] Establish a world coordinate system and specify a three-dimensional coordinate for each corner of the checkerboard in the world coordinate system.

[0058] Step 1.4: Using the two-dimensional positions of the corner points on the calibration image and their corresponding three-dimensional coordinates, calculate the internal and external parameters of the camera using a camera calibration algorithm.

[0059] Solve for the camera parameters. Using the two-dimensional positions of the corner points in the image and their corresponding three-dimensional world coordinates, calculate the internal and external parameters of the camera using a camera calibration algorithm (such as Zhang's calibration method). The internal parameters include: focal length f, optical center (c x , c y ), and distortion coefficients (such as radial and tangential distortion coefficients). The external parameters include: rotation matrix R and translation vector t, which describe the position and orientation of the camera relative to a certain world coordinate system.

[0060] Step 1.5: Optimize the internal and external parameters of the camera. The goal is to minimize the reprojection error to obtain the optimized internal and external parameters.

[0061] To further improve the calibration accuracy and reduce the reprojection error, it is necessary to optimize the internal and external parameters of the camera. The goal is to minimize the reprojection error, that is, to minimize the difference between the actually detected corner point positions and the corner point positions calculated according to the camera model.

[0062] Minimize the reprojection error, which is expressed by the formula:

[0063] e = ∑(p obs - p proj ) 2 (1)

[0064] where e represents the reprojection error, and p obs and p proj represent the observed point position and the projected point position calculated through the camera parameters, respectively.

[0065] Furthermore, p proj = K[R|t]P (2)

[0066] where P represents the three-dimensional world point; K represents the internal parameter matrix of the camera, including the focal length and optical center, etc.; R represents the rotation matrix of the camera, reflecting the rotation of the camera relative to the world coordinate system; t represents the translation vector of the camera, indicating the position of the camera relative to the world coordinate system.

[0067] Step 1.6: Use the optimized internal and external parameters to perform distortion correction and three-dimensional reconstruction on the new calibration image to verify the accuracy of the calibration result.

[0068] Using the obtained camera parameters, especially the distortion coefficients, perform distortion correction on the collected images to remove the distortion introduced by the camera lens. Calculate the reprojection error of each corner point using formula (1), and this error can be used to evaluate the quality of the calibration.

[0069] Step 1.7: If the accuracy of verifying the calibration result does not meet the requirements, recalibrate until the accuracy of the calibration result meets the requirements, and use the optimized internal and external parameters at this time as the initial parameters of the camera.

[0070] Use the internal and external parameters of the camera obtained by calibration to perform distortion correction and three-dimensional reconstruction on the new image to verify the accuracy of the calibration. If the calibration accuracy is not high and does not meet the requirements, recalibration is required. During the recalibration process, the position and angle of the calibration object can be changed to obtain more or higher-quality calibration images, or the parameters of image processing and corner detection can be adjusted to improve the detection accuracy of corners and the accuracy of calibration. Until the accuracy of the calibration result meets the requirements, use the optimized internal and external parameters at this time as the initial parameters of the camera.

[0071] Step 2: Use the camera to take pictures of the human body, and perform initial parameter optimization and three-dimensional reconstruction on the camera for the human body area to obtain a sparse point cloud of the human body area.

[0072] Use the camera to take pictures of the human body. Each time a picture is taken, run a feature point operator extraction and an image-based human body area MASK segmentation once, and then use local bundle_adjustment to optimize the camera parameters for the human body area to improve the accuracy, and at the same time obtain a sparse point cloud of the human body.

[0073] The specific steps of Step 2 include:

[0074] Step 2.1: Use the camera to take pictures of the human body, perform feature point extraction and human body area MASK segmentation on the two-dimensional images taken each time, extract the feature points in the two-dimensional images, and segment the human body area.

[0075] During the shooting process, run a feature point detection algorithm on each taken image to identify unique feature points in the image. These feature points should be able to be repeatedly identified from multiple perspectives for subsequent matching and three-dimensional reconstruction processes.

[0076] Use a deep learning model, such as Mask R-CNN, to segment the image to identify and separate the human body area. The purpose is to generate the MASK of the human body area in each image, so that the human body features can be focused on in the subsequent steps, improving the processing accuracy and efficiency.

[0077] Step 2.2: Use a feature point matching algorithm to perform feature point matching between different two-dimensional images, and use the matched feature points and the initial parameters of the camera, and use a three-dimensional reconstruction algorithm to estimate the position of each feature point in three-dimensional space to obtain a sparse point cloud of the human body area.

[0078] Specifically, using the feature point matching algorithm, the same feature points among different images are found. The key to this step lies in identifying the same physical point observed in multiple images.

[0079] Further, perform the sparse three-dimensional reconstruction process. Utilize the matched feature points and the initial parameters of the camera (obtained through camera calibration), and use a three-dimensional reconstruction algorithm (such as multi-view geometry or Structure from Motion SfM algorithm) to estimate the position of each feature point in the three-dimensional space, obtaining a sparse point cloud of the human body region.

[0080] Step 2.3: Run the local Bundle Adjustment to optimize the initial parameters of the camera and the coordinates of the sparse point cloud of the human body region, in order to minimize the reprojection error, and obtain the locally optimized camera parameters and sparse point cloud.

[0081] After obtaining the preliminary three-dimensional reconstruction sparse point cloud and camera parameters, run the local Bundle Adjustment (BA) to optimize the internal and external parameters of the camera. In this process, focus on the feature points within the human MASK region. By minimizing the reprojection error, that is, minimizing the difference between the observed position of the feature point and the predicted image position based on the current camera parameters and the three-dimensional point position, adjust the camera parameters and the point cloud coordinates.

[0082] The local bundle adjustment optimizes the camera parameters and the 3D point coordinates in the scene to minimize the reprojection error, which can be expressed as:

[0083]

[0084] where, C i and X j represent the parameters of the i-th camera and the 3D coordinates of the j-th point in the scene respectively; π is the projection function that maps the 3D point X j to the 2D image point using the camera parameters C i . p ij is the observed image point of the j-th point in the i-th camera. w ij is the weight coefficient, indicating the confidence or importance of the i-th camera observing the j-th point. C represents the set of camera parameters, including internal and external parameters; X represents the set of 3D point coordinates in the scene; n is the total number of observed points in the image; m is the total number of 3D points in the scene.

[0085] Compared with the global Bundle Adjustment, the local BA adopted in the present invention pays more attention to a specific region (such as the human body region), and adopts a hierarchical or step-by-step optimization strategy to improve the computational efficiency and the accuracy of parameter optimization.

[0086] The optimized reprojection error is calculated by formula (3) to evaluate the quality of the optimization process and ensure that the error is small enough. As needed, various steps of the algorithm may be adjusted, such as improving the stability of feature point detection, adjusting the accuracy of MASK segmentation, or adjusting the parameters of Bundle Adjustment to further improve the accuracy and quality of the reconstruction.

[0087] Finally, this process generates sparse point clouds for the human body region, which can be used for further analysis, such as human pose estimation, motion analysis, or further dense reconstruction.

[0088] Step 3: Reconstruct the dense point cloud of the human object based on the sparse point cloud of the human body region.

[0089] According to the human segmentation result, the human body region to be reconstructed on the image can be obtained; through the multi-view stereo geometry method and the locally optimized camera parameters, the depth information of the human body region can be obtained; further, by fusing the depth information with the sparse point cloud of the human body region, a dense point cloud representing the human object can be generated.

[0090] The specific steps of step 3 include:

[0091] Step 3.1: Human body region segmentation. According to the results of human body segmentation of each frame of image by Mask R-CNN, the human body region in the image is identified.

[0092] Step 3.2: Multi-view geometry and depth estimation, including triangulation and multi-view stereo matching steps.

[0093] Triangulation: For the feature points matched in multiple images, using their pixel positions and the locally optimized internal and external camera parameters, calculate the position of each point in the three-dimensional space through the triangulation method to obtain the depth information.

[0094] Multi-view stereo matching: By comparing the similarity of the human body region in different views, the corresponding relationship between pixels can be estimated at a finer level, so as to obtain the depth information of each pixel. This step can improve the accuracy of depth estimation through global optimization methods.

[0095] Step 3.3: Depth information fusion and dense point cloud reconstruction.

[0096] The depth map fusion process is to fuse the depth maps obtained from different perspectives into a unified three-dimensional model. The depth information is calculated from multiple images, and its mathematical model can be simplified to the depth calculation formula:

[0097]

[0098] Where Z represents the depth of the point relative to the camera; f is the focal length of the camera; B is the baseline distance between the two cameras in the stereo setup; d is the horizontal position difference (disparity) of the features in the two stereo images.

[0099] Dense point cloud generation: Through the depth information, a dense point cloud representing the human body object can be generated. Each point in the point cloud represents a specific position on the human body surface, and the density of the point cloud is sufficient to describe the shape and features of the human body in detail.

[0100] The dense point cloud of the human body object reconstructed in step 3 is obtained by further processing based on the sparse point cloud obtained in step 2. In step 2, a sparse point cloud is obtained through feature point extraction, matching, and three-dimensional reconstruction, which provides a preliminary three-dimensional structure for obtaining depth information. Step 3 further obtains more detailed depth information through multi-view geometry methods and depth estimation on this basis, thereby generating a dense point cloud representing the human body. Thus, in step 3, based on the sparse point cloud in step 2, the density and details of the point cloud are increased through depth estimation and information fusion.

[0101] Step 4: Reconstruct a watertight human body surface mesh based on the dense point cloud of the human body object.

[0102] Use the parametric human SMPLX model as an initial solution and as a global regularization term to constrain the mesh generation step to reconstruct a watertight human body surface mesh. The specific steps of step 4 include:

[0103] Step 4.1: Extract the SMPLX model from the image and perform an initial match with the dense point cloud of the human body object.

[0104] First, extract the SMPLX model parameters from the captured image, which is completed by combining global search and local optimization. Using the dense point cloud of the human body object as a reference, the SMPLX model generated by these parameters is made to fit the human body shape in the image as closely as possible. Then, use this SMPLX model as the initial value to ensure that the generated mesh can more accurately represent the human body shape.

[0105] Step 4.2: Make the generated mesh fit the dense point cloud and maintain the geometric characteristics of the SMPLX model through global minimization of the geometric error.

[0106] After the initial matching, an error function is defined to quantify the difference between the mesh to be generated and the dense point cloud. This error is typically the sum of the distances from the points in the dense point cloud to the closest points on the surface of the generated mesh. During the mesh construction process, the SMPLX model is used as a global regularization term to constrain the shape of the mesh, such that the generated mesh not only fits well with the dense point cloud but also maintains the geometric characteristics of the SMPLX model. By adjusting the shape and position parameters of the mesh to be generated, the gradient descent method is used to minimize the above error function while considering the regularization term of the SMPLX model. This process needs to be iterated multiple times until a predetermined convergence criterion is reached.

[0107] Step 4.3: Perform line-of-sight conflict detection and resolve line-of-sight conflicts to obtain the human body model after global optimization.

[0108] Line-of-sight conflict detection: During the optimization process, special attention needs to be paid to the issue of line-of-sight conflicts, that is, some parts of the model incorrectly occlude parts that should be visible. This can be detected by simulating observations of the model from multiple perspectives to ensure that the lines of sight of all observed point clouds can find corresponding visible surfaces on the model.

[0109] Conflict resolution: Once a line-of-sight conflict is detected, it can be resolved by adjusting the pose or shape parameters of the model.

[0110] Step 4.4: Refine and seal the generated mesh to reconstruct a watertight human body surface mesh.

[0111] After global optimization is completed, the obtained mesh parameters can generate a human body model with a high degree of shape conformity to the dense point cloud. Then the model can be further refined to ensure that the generated human body mesh is watertight, that is, there are no holes. Further refining the model means increasing the mesh density of the model, adding more vertices and faces to more precisely describe the shape and details of the human body. This includes adding more meshes in areas that require higher details and adjusting the layout of the existing meshes to better match the data in the dense point cloud.

[0112] Step 5: Optimize the human body surface mesh to restore a three-dimensional model with more human body details.

[0113] Optimize the human body surface mesh. On the premise of ensuring the two-dimensional to three-dimensional consistency of human body joint points and facial key points, minimize the reprojection error of the three-dimensional model on different two-dimensional images as much as possible, and at the same time perform the optimization of the structural prior to restore more human body details, obtain the three-dimensional model of the human body object, and ensure the surface accuracy and model compactness of the reconstructed human body object.

[0114] The specific steps of step 5 include:

[0115] Step 5.1: Align 2D and 3D key points by extracting 2D key points and performing 3D reconstruction.

[0116] Extract 2D key points: Extract 2D key points of the human body and face from each captured image, which can be achieved through existing human pose estimation and face recognition algorithms.

[0117] Establish correspondence: Determine the corresponding key points of each 2D key point in the 3D model.

[0118] 3D reconstruction: Use the principles of multi-view geometry, combined with the internal and external parameters of the camera, to perform 3D reconstruction on the 2D key points to obtain their positions in 3D space.

[0119] Step 5.2: Minimize the reprojection error.

[0120] Calculate the reprojection error: For each 3D key point, calculate its expected position on each image through the camera model, and the difference between this and the position of the actually detected 2D key point is the reprojection error.

[0121] Optimize the model parameters: Adjust the parameters of the 3D model (including shape, pose, etc.) to minimize the total reprojection error of all key points.

[0122] Adjust the mesh to fit the 3D reconstruction result as closely as possible while minimizing geometric errors and reprojection errors, which can be expressed as:

[0123]

[0124] where V and F represent the vertices and faces of the mesh respectively; P i is the 2D key point observed in image i; Φ is a function that projects 3D mesh points onto the 2D image plane according to the camera parameters C i ; Ψ is a regularization term used to add prior knowledge about human shape and pose to ensure that the mesh remains realistic; λ is a weight factor that balances the importance of data fidelity and the regularization term; k represents the number of observed 2D key points. This means that during the optimization process, the reprojection errors of all key points are considered to ensure the accuracy and consistency of the 3D model from multiple viewpoints.

[0125] Step 5.3: Application of structural priors.

[0126] Introduce structural prior knowledge: Use prior knowledge, i.e., the anatomical structure and motion characteristics of the human body, to guide the adjustment of the model and help restore human details in the absence of sufficient image information.

[0127] Detail restoration: Utilize the structural prior to refine the local details of the model, such as facial expressions or finger postures, while maintaining overall consistency and naturalness, thereby restoring a 3D model with more human body details.

[0128] Step 6: Visualize and render the 3D model using texture mapping to generate a 3D human body model.

[0129] Map the human body color information in the 2D image to the texture map according to the corresponding relationship of solid geometry and globally optimize the map to reduce the problem of uneven skin color caused by color difference changes in different images; Apply the optimized texture map to the 3D model and use graphics rendering technology for visualization rendering to generate a 3D human body model.

[0130] The specific steps of Step 6 include:

[0131] Step 6.1: Generate a texture map by extracting color information and performing texture mapping.

[0132] Extract color information: Extract the color information of the human body from each captured image, including color correction and brightness adjustment, to ensure the consistency of the color information extracted from different images.

[0133] Establish the corresponding relationship: Determine the corresponding positions of each image pixel on the 3D model through multi-view stereo matching, and use the internal and external parameters of the camera to calculate the 3D space points corresponding to each pixel point on the image.

[0134] Generate a texture map: Map the extracted color information to the surface of the 3D model to generate a texture map.

[0135] Texture coordinate mapping: Assign texture coordinates to each vertex on the model to ensure that the texture image is correctly covered on the 3D model.

[0136] Step 6.2: Globally optimize the map, including performing color difference correction, texture fusion, and optimization algorithms.

[0137] Color difference correction: To reduce the problem of uneven skin color caused by color difference changes between different images, color difference correction is required, which is achieved through a global color correction algorithm.

[0138] Texture fusion: In the overlapping areas on the model surface, the textures of different images may produce edges or discontinuities. Through texture fusion technology, these transition areas can be smoothed to reduce visual discontinuities.

[0139] Optimization Algorithm: The process of globally optimizing texture mapping can be achieved by minimizing an energy function that quantifies the color differences between textures and the smoothness of transition regions. The optimization process can utilize gradient descent, genetic algorithms, or other optimization techniques to find the minimum value of the energy function.

[0140] Step 6.3: Apply the optimized texture map to the 3D model and use graphics rendering technology for visual rendering to generate the final 3D human body model. This step can present a highly realistic human body model with rich details and natural and consistent colors.

[0141] An automated 3D human body model reconstruction method based on explicit space proposed by the present invention takes into account the human body details and texture fineness of the reconstructed human body model, and realizes the high fineness of the human body model through steps such as continuously optimizing camera parameters, reducing reprojection errors, globally minimizing geometric errors, and global texture mapping. Compared with the prior art, the main advantages of the present invention are: 1) The present invention forms an automated 3D human body model reconstruction method based on the human body explicit space by using multi-angle human body image data; 2) Geometric errors can be better reduced and camera parameters can be optimized during the modeling process; 3) The optimized reconstruction integrates the human body surface mesh with the depth information of the human body region, ensuring the surface accuracy and model compactness of the reconstructed human body; 4) A more refined human body reconstruction model can be obtained.

[0142] In some embodiments, the present invention also provides a computer program product, including a computer program, which when executed by a processor, implements the steps of the automated 3D human body model reconstruction method based on explicit space.

[0143] In some embodiments, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the automated 3D human body model reconstruction method based on explicit space.

[0144] In some embodiments, the present invention further provides a computer device, which includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store transactions to be processed. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it can implement the steps of the method for automatically reconstructing a three-dimensional human body model based on explicit space.

[0145] In the present invention, specific examples are used to elaborate on the principles and implementation manners of the present invention. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An automated reconstruction method for a three-dimensional human body model based on explicit space, characterized in that, Including: Calibrate the internal and external parameters of the camera according to the chessboard grid calibration method to obtain the initial parameters of the camera; Use the camera to take pictures of the human body, and optimize the initial parameters of the camera and perform 3D reconstruction on the human body area to obtain the sparse point cloud of the human body area; Reconstruct the dense point cloud of the human object based on the sparse point cloud of the human body area; According to the human body segmentation result, obtain the human body area to be reconstructed on the image; through the multi-view stereo geometry method and the locally optimized camera parameters, obtain the depth information of the human body area; further fuse the depth information with the sparse point cloud of the human body area to generate a dense point cloud representing the human object; The reconstruction of the dense point cloud of the human object based on the sparse point cloud of the human body area specifically includes: Step 3.1: Human body area segmentation. According to the result of human body segmentation of each frame of image by Mask R-CNN, identify the human body area in the image; Step 3.2: Multi-view geometry and depth estimation, including triangulation and multi-view stereo matching steps; Triangulation: For the feature points matched in multiple images, use their pixel positions and the locally optimized internal and external camera parameters to calculate the position of each point in the three-dimensional space through the triangulation method, so as to obtain the depth information; Multi-view stereo matching: By comparing the similarity of the human body area in different views, estimate the correspondence between pixels at a finer level, so as to obtain the depth information of each pixel; improve the accuracy of depth estimation through the global optimization method; Step 3.3: Depth information fusion and dense point cloud reconstruction; The depth map fusion process is to fuse depth maps obtained from different perspectives into a unified three-dimensional model; depth information is calculated from multiple images, and its mathematical model can be simplified into a depth calculation formula: where Z represents the depth of the point relative to the camera; f is the focal length of the camera; B is the baseline distance between the two cameras in the stereo setup; d is the horizontal position difference of the features in the two stereo images; Dense point cloud generation: Generate a dense point cloud representing the human object through the depth information; each point in the point cloud represents a specific position on the human body surface, and the density of the point cloud is sufficient to describe the shape and features of the human body in detail Reconstruct a watertight human body surface mesh based on the dense point cloud of the human object, specifically including: Step 4.1: Extract the SMPLX model from the image and perform an initial match with the dense point cloud of the human object; First, extract the SMPLX model parameters from the captured image, which is completed by combining global search and local optimization. Using the dense point cloud of the human object as a reference, make the SMPLX model generated by these parameters as close as possible to the human body shape in the image; then use this SMPLX model as the initial value to ensure that the generated mesh can more accurately represent the human body shape; Step 4.2: Make the generated mesh coincide with the dense point cloud and maintain the geometric characteristics of the SMPLX model by globally minimizing the geometric error; After the initial match, define an error function to quantify the difference between the mesh to be generated and the dense point cloud. This error is the sum of the distances from the points in the dense point cloud to the nearest points on the surface of the generated mesh; during the mesh generation process, use the SMPLX model as a global regularization term to constrain the shape of the mesh, so that the generated mesh not only coincides with the dense point cloud, but also maintains the geometric characteristics of the SMPLX model; by adjusting the shape and position parameters of the mesh to be generated, use the gradient descent method to minimize the above error function, while considering the regularization term of the SMPLX model; this process needs to be iterated multiple times until a predetermined convergence criterion is reached; Step 4.3: Perform line-of-sight conflict detection and resolve line-of-sight conflicts to obtain the human body model after global optimization; Line-of-sight conflict detection: During the optimization process, detect line-of-sight conflicts by simulating the observation of the model from multiple perspectives to ensure that the line of sight of all observed point clouds can find corresponding visible surfaces on the model; Conflict resolution: Once a line-of-sight conflict is detected, resolve it by adjusting the pose or shape parameters of the model; Step 4.4: Refine and seal the generated mesh to reconstruct a watertight human body surface mesh; After global optimization is completed, the obtained mesh parameters can generate a human body model with a high degree of shape coincidence with the dense point cloud; then further refine the model to ensure that the generated human body mesh is watertight, that is, without holes; further refining the model means increasing the mesh density of the model and adding more vertices and faces; this includes adding more meshes in areas that require higher details and adjusting the layout of the existing meshes to better match the data in the dense point cloud; Optimize the human body surface mesh to restore a three-dimensional model with more human body details; Use texture mapping to perform visual rendering on the three-dimensional model to generate a three-dimensional human body model.

2. The automated reconstruction method of a three-dimensional human body model based on explicit space according to claim 1, wherein Calibrate the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial parameters of the camera, specifically including: Select a checkerboard and use the camera to collect calibration images of the checkerboard at different positions and angles; Use a corner detection algorithm to find the corners of the checkerboard in each calibration image and refine the positions of these corners through sub-pixel corner detection technology; Specify a three-dimensional coordinate for each corner of the checkerboard in the world coordinate system; Use the two-dimensional position of the corner in the calibration image and the corresponding three-dimensional coordinates, and use the camera calibration algorithm to calculate the internal and external parameters of the camera; Optimize the internal and external parameters of the camera, with the goal of minimizing the reprojection error, to obtain the optimized internal and external parameters; Use the optimized internal and external parameters to perform distortion correction and three-dimensional reconstruction on the new calibration image to verify the accuracy of the calibration result; If the accuracy of verifying the calibration result does not meet the requirements, re-perform calibration until the accuracy of the calibration result meets the requirements, and use the optimized internal and external parameters at this time as the initial parameters of the camera.

3. The automated three-dimensional human body model reconstruction method based on explicit space according to claim 2, wherein Use the camera to take pictures of the human body, and perform initial parameter optimization and three-dimensional reconstruction of the camera for the human body area to obtain a sparse point cloud of the human body area, specifically including: Use the camera to take pictures of the human body, extract feature points and perform human body area MASK segmentation on each captured two-dimensional image, extract the feature points in the two-dimensional image and segment the human body area; Use a feature point matching algorithm to perform feature point matching between different two-dimensional images, and use the matched feature points and the initial parameters of the camera, and use a three-dimensional reconstruction algorithm to estimate the position of each feature point in three-dimensional space to obtain a sparse point cloud of the human body area; Run local Bundle Adjustment to optimize the initial parameters of the camera and the coordinates of the sparse point cloud of the human body area to minimize the reprojection error and obtain the locally optimized camera parameters and sparse point cloud.

4. The automatic reconstruction method of a three-dimensional human body model based on explicit space according to claim 3, wherein, Optimize the human body surface mesh to restore a three-dimensional model with more human body details, specifically including: Optimize the human body surface mesh. On the premise of ensuring the two-dimensional to three-dimensional consistency of human joint points and facial key points, minimize the reprojection error of the three-dimensional model on different two-dimensional images as much as possible. At the same time, optimize the structural prior to restore more human body details and obtain a three-dimensional model of the human body object.

5. The automated reconstruction method of a three-dimensional human body model based on explicit space according to claim 4, characterized in that, Visualize and render the three-dimensional model using texture mapping to generate a three-dimensional human body model, specifically including: Map the human body color information in the two-dimensional image to the texture mapping according to the corresponding relationship of solid geometry and globally optimize the mapping; Apply the optimized texture mapping to the three-dimensional model and use graphics rendering technology for visualization rendering to generate a three-dimensional human body model.

6. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method for automatically reconstructing a three-dimensional human body model based on explicit space according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method for automatically reconstructing a three-dimensional human body model based on explicit space according to any one of claims 1-5.

8. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for automatically reconstructing a three-dimensional human body model based on explicit space according to any one of claims 1-5.

Images