Human body three-dimensional model automatic reconstruction method based on explicit space, product, medium and equipment
Through the automated reconstruction method of three-dimensional human body model based on explicit space, the problems of poor geometric structure and blurred edges in the existing technology are solved, and high-precision human body model reconstruction is realized, meeting the needs of high-precision application.
Patent Information
- Application Number
- CN202411060915.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-02
AI Technical Summary
The existing three-dimensional human body model reconstruction technology based on hidden space has problems of poor geometric structure and blurred edges, which cannot meet the needs of high precision.
The automated reconstruction method of the three-dimensional human body model based on explicit space is adopted, and the camera parameters are calibrated through checkerboard grids, the camera initial parameters are optimized, dense point clouds and watertight grids are generated, and visual rendering is combined with texture maps to ensure the fineness of human body details and textures.
High-precision three-dimensional model reconstruction of human body is realized, geometric errors and reprojection errors are reduced, and a more refined human body model is obtained, meeting the needs of high-precision applications.
Smart Images

Figure CN118967974A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional modeling, and in particular to a method, product, medium and equipment for automatically reconstructing a three-dimensional model of a human body based on explicit space. Background Art
[0002] In the 1960s and 1970s, computer graphics began to emerge. Early research focused on graphics rendering, modeling, and animation. Later, people began to try to use computers to simulate and present three-dimensional objects. With the progress of computer graphics, three-dimensional modeling technology gradually emerged. Researchers began to try to digitize the shape, texture and other information of actual objects and store them in the form of three-dimensional models in computers. In the 1980s and 1990s, significant progress was made in the field of computer vision. Researchers began to focus on how to make computer systems understand and interpret images. This laid the foundation for extracting three-dimensional information from two-dimensional images, and also provided technical support for the development of three-dimensional model reconstruction technology for the human body.
[0003] With the continuous breakthroughs in sensor technology, the continuous optimization of algorithms, and the development of deep learning technology in recent years, the accuracy of human 3D models has been continuously improved. Currently, the application of human 3D models is very extensive. In medicine, this technology can be applied to disease diagnosis, surgical planning, medical research, etc.; in entertainment, this technology can be applied to animation production, character design, game modeling, etc.; in sports training, this technology can be applied to professional athletes' sports analysis, etc. The development and progress of human 3D model reconstruction technology will also bring more possibilities for its application fields, so higher requirements are put forward for the accuracy of this technology.
[0004] The current mainstream digital human modeling method is based on latent space, such as nerfneus. The overall color of the model is good, but the geometric structure is poor and there is a problem of blurred edges, which cannot meet the actual needs in some application scenarios. Therefore, how to break through the current human 3D model reconstruction method and explore higher-precision modeling methods and systems has become an urgent problem for researchers in this field. Summary of the invention
[0005] The purpose of the present invention is to provide a method, product, medium and equipment for automatically reconstructing a three-dimensional human body model based on explicit space, which can automatically reconstruct a three-dimensional human body model according to the explicit space of the human body and ensure the fineness of human body details and textures.
[0006] To achieve the above object, the present invention provides the following solution.
[0007] On the one hand, the present invention provides a method for automatically reconstructing a three-dimensional human body model based on an explicit space, comprising:
[0008] Calibrate the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial parameters of the camera;
[0009] Use a camera to shoot the human body, optimize the initial camera parameters and perform 3D reconstruction for the human body area to obtain a sparse point cloud of the human body area;
[0010] Reconstructing dense point cloud of human object based on sparse point cloud of human region;
[0011] Reconstruct a watertight human body surface mesh based on the dense point cloud of the human body object;
[0012] Optimize the human body surface mesh to restore a 3D model with more human body details;
[0013] The texture map is used to visualize the 3D model and generate a 3D human body model.
[0014] Optionally, calibrating the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial parameters of the camera specifically includes:
[0015] Select a chessboard and use a camera to collect calibration images of the chessboard at different positions and angles;
[0016] Use the corner detection algorithm to find the corners of the checkerboard in each calibration image and refine the positions of these corners using sub-pixel corner detection technology;
[0017] Assign a three-dimensional coordinate in the world coordinate system to each corner point of the chessboard;
[0018] The camera calibration algorithm is used to calculate the camera's internal and external parameters using the 2D positions of the corner points on the calibration image and the corresponding 3D coordinates.
[0019] Optimize the internal and external parameters of the camera to minimize the reprojection error and obtain the optimized internal and external parameters;
[0020] Use the optimized internal and external parameters to perform distortion correction and 3D reconstruction on the new calibration image to verify the accuracy of the calibration result;
[0021] If the accuracy of the calibration result does not meet the requirements, recalibration is performed until the accuracy of the calibration result meets the requirements, and the optimized internal and external parameters are used as the initial parameters of the camera.
[0022] Optionally, photographing a human body with a camera, optimizing initial camera parameters and performing three-dimensional reconstruction on a human body region to obtain a sparse point cloud of the human body region specifically includes:
[0023] Use the camera to shoot human body, extract feature points and perform human body area MASK segmentation on the two-dimensional image taken each time, extract feature points in the two-dimensional image and segment the human body area;
[0024] Use feature point matching algorithm to match feature points between different two-dimensional images, and use the matched feature points and the initial parameters of the camera to use a three-dimensional reconstruction algorithm to estimate the position of each feature point in three-dimensional space to obtain a sparse point cloud of the human body area;
[0025] Run local BundleAdjustment to optimize the initial camera parameters and the sparse point cloud coordinates of the human body area to minimize the reprojection error and obtain the locally optimized camera parameters and sparse point cloud.
[0026] Optionally, reconstructing a dense point cloud of a human object based on a sparse point cloud of a human body region specifically includes:
[0027] The depth information of the human body area is obtained through multi-view stereo geometry method and locally optimized camera parameters;
[0028] The sparse point cloud of the human body area is fused with depth information to generate a dense point cloud representing the human object.
[0029] Optionally, reconstructing a watertight human body surface mesh based on a dense point cloud of a human body object specifically includes:
[0030] Extract the SMPLX model from the captured image and perform an initial match with the dense point cloud of the human object;
[0031] By globally minimizing the geometric error, the generated mesh is consistent with the dense point cloud and the geometric characteristics of the SMPLX model are maintained;
[0032] Perform sight conflict detection and resolve sight conflict to obtain a human body model after global optimization;
[0033] The human body model mesh is refined and sealed to reconstruct a watertight human body surface mesh.
[0034] Optionally, the optimizing the human body surface mesh to restore a three-dimensional model with more human body details specifically includes:
[0035] Optimize the human body surface mesh, and minimize the reprojection error of the three-dimensional model on different two-dimensional images while ensuring the two-dimensional to three-dimensional consistency of the human body joints and facial key points. At the same time, optimize the structural prior to restore more human body details and obtain a three-dimensional model of the human body object.
[0036] Optionally, the three-dimensional model is visually rendered using a texture map to generate a three-dimensional human body model, specifically including:
[0037] Map the human body color information in the two-dimensional image to the texture map according to the stereoscopic geometric correspondence, and globally optimize the texture;
[0038] The optimized texture map is applied to the three-dimensional model, and the visual rendering is performed using graphics rendering technology to generate a three-dimensional model of the human body.
[0039] On the other hand, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for automatic reconstruction of a three-dimensional human body model based on explicit space.
[0040] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for automatically reconstructing a three-dimensional human body model based on explicit space.
[0041] On the other hand, the present invention also provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for automatic reconstruction of a three-dimensional human body model based on explicit space.
[0042] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0043] The present invention takes into account the human body details and texture refinement of the reconstructed human body model, and achieves high precision of the human body model by continuously optimizing camera parameters, reducing reprojection errors, globally minimizing geometric errors, and global mapping. Specifically, the present invention forms an automated human body three-dimensional model reconstruction method based on human body explicit space by utilizing multi-angle human body image data, which can better reduce geometric errors and optimize camera parameters during the modeling process. In addition, the present invention optimizes and reconstructs the human body surface mesh that integrates the depth information of the human body region, ensures the surface accuracy and model compactness of the reconstructed human body, and can obtain a more refined human body reconstruction model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 The present invention provides a flowchart of a method for automatically reconstructing a three-dimensional human body model based on explicit space. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] The purpose of the present invention is to provide a method, product, medium and equipment for automatically reconstructing a three-dimensional human body model based on explicit space, so as to automatically reconstruct the three-dimensional human body model according to the explicit space of the human body while ensuring the fineness of human body details and textures.
[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0049] See also Figure 1 The present invention provides a method for automatically reconstructing a three-dimensional human body model based on an explicit space, comprising:
[0050] Step 1: Calibrate the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial parameters of the camera.
[0051] The present invention calibrates the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial values of the camera parameters. The step 1 specifically includes:
[0052] Step 1.1: Select a checkerboard and use a camera to collect calibration images of the checkerboard at different positions and angles.
[0053] Use a chessboard with obvious black and white contrast as a calibration object. Place the chessboard at different positions and angles, and use a camera to take pictures of the chessboard to obtain enough image data.
[0054] Step 1.2: Use the corner detection algorithm to find the corners of the checkerboard in each calibration image and refine the positions of these corners using sub-pixel corner detection technology.
[0055] In each captured image, a corner detection algorithm is used to find the inner corners of the chessboard. The positions of these corners are further refined through sub-pixel corner detection technology to improve the accuracy of calibration.
[0056] Step 1.3: Assign a three-dimensional coordinate in the world coordinate system to each corner point of the chessboard.
[0057] A world coordinate system is established, and a three-dimensional coordinate is assigned to each corner point of the chessboard in the world coordinate system.
[0058] Step 1.4: Using the 2D positions of the corner points on the calibration image and the corresponding 3D coordinates, use the camera calibration algorithm to calculate the camera's intrinsic and extrinsic parameters.
[0059] Solve the camera parameters, use the two-dimensional position of the corner points on the image and their corresponding three-dimensional world coordinates, and use the camera calibration algorithm (such as Zhang Zhengyou calibration method) to calculate the internal and external parameters of the camera. The internal parameters include: focal length f, optical center (c x , c y ), and distortion coefficients (such as radial distortion and tangential distortion coefficients). The external parameters include: rotation matrix R and translation vector t, which describe the position and attitude of the camera relative to a certain world coordinate system.
[0060] Step 1.5: Optimize the internal and external parameters of the camera. The goal is to minimize the reprojection error and obtain the optimized internal and external parameters.
[0061] In order to further improve the calibration accuracy and reduce the reprojection error, it is necessary to optimize the internal and external parameters of the camera. The goal is to minimize the reprojection error, that is, to minimize the difference between the actual detected corner point position and the corner point position calculated according to the camera model.
[0062] Minimize the reprojection error, the formula is expressed as:
[0063] e=∑(p obs -p proj ) 2 (1)
[0064] Where e represents the reprojection error, p obs and p proj They represent the observed point position and the projected point position calculated by the camera parameters respectively.
[0065] Furthermore, p proj =K[R|t]P(2)
[0066] Where P represents a three-dimensional world point; K represents the intrinsic parameter matrix of the camera, including focal length and optical center; R represents the rotation matrix of the camera, which reflects the rotation of the camera relative to the world coordinate system; t represents the translation vector of the camera, indicating the position of the camera relative to the world coordinate system.
[0067] Step 1.6: Use the optimized internal and external parameters to perform distortion correction and 3D reconstruction on the new calibration image to verify the accuracy of the calibration result.
[0068] The acquired image is distorted using the obtained camera parameters, especially the distortion coefficients, to remove the distortion introduced by the camera lens. The reprojection error of each corner point is calculated using formula (1), which can be used to evaluate the quality of the calibration.
[0069] Step 1.7: If the accuracy of the calibration result does not meet the requirements, recalibrate until the accuracy of the calibration result meets the requirements, and use the optimized internal and external parameters as the initial parameters of the camera.
[0070] Use the camera's internal and external parameters obtained through calibration to perform distortion correction and 3D reconstruction on the new image to verify the accuracy of the calibration. If the calibration accuracy is not high and does not meet the requirements, recalibration is required. During the recalibration process, the position and angle of the calibration object can be changed to obtain more or higher quality calibration images, or the parameters of image processing and corner detection can be adjusted to improve the accuracy of corner detection and calibration. When the accuracy of the calibration result meets the requirements, the optimized internal and external parameters are used as the initial parameters of the camera.
[0071] Step 2: Use a camera to shoot the human body, optimize the initial camera parameters and perform 3D reconstruction for the human body area to obtain a sparse point cloud of the human body area.
[0072] Use the camera to shoot human bodies. Each time you shoot, run a feature point operator extraction and an image-based human body area MASK segmentation. Then use local bundle_adjustment to optimize the camera parameters for the human body area to improve accuracy and obtain a sparse point cloud of the human body.
[0073] The step 2 specifically includes:
[0074] Step 2.1: Use a camera to shoot the human body, extract feature points and perform human body area MASK segmentation on the two-dimensional image taken each time, extract feature points in the two-dimensional image and segment the human body area.
[0075] During the shooting process, a feature point detection algorithm is run on each image to identify unique feature points in the image. These feature points should be able to be repeatedly identified in multiple perspectives for subsequent matching and 3D reconstruction.
[0076] Use deep learning models, such as Mask R-CNN, to segment images to identify and separate human regions. The goal is to generate a mask of the human region in each image, so that you can focus on human features in subsequent steps and improve processing accuracy and efficiency.
[0077] Step 2.2: Use the feature point matching algorithm to match the feature points between different two-dimensional images, and use the matched feature points and the initial parameters of the camera to use the three-dimensional reconstruction algorithm to estimate the position of each feature point in the three-dimensional space to obtain a sparse point cloud of the human body area.
[0078] Specifically, a feature point matching algorithm is used to find the same feature points between different images. The key to this step is to identify the same physical point observed in multiple images.
[0079] Furthermore, a sparse 3D reconstruction process is performed, using the matched feature points and the initial parameters of the camera (obtained through camera calibration), and a 3D reconstruction algorithm (such as multi-view geometry or structure from motion (SfM) algorithm) is used to estimate the position of each feature point in the 3D space to obtain a sparse point cloud of the human body area.
[0080] Step 2.3: Run local BundleAdjustment to optimize the initial camera parameters and the sparse point cloud coordinates of the human body area to minimize the reprojection error and obtain the locally optimized camera parameters and sparse point cloud.
[0081] After obtaining the preliminary 3D reconstructed sparse point cloud and camera parameters, run the local Bundle Adjustment (BA) to optimize the camera's internal and external parameters. In this process, we focus on the feature points in the human body MASK area and adjust the camera parameters and point cloud coordinates by minimizing the reprojection error, that is, minimizing the difference between the observed position of the feature point and the image position predicted based on the current camera parameters and 3D point position.
[0082] Local bundle adjustment optimizes the camera parameters and the 3D point coordinates in the scene to minimize the reprojection error, which can be expressed as:
[0083]
[0084] Among them, C i and X j represents the parameters of the i-th camera and the 3D coordinates of the j-th point in the scene; π is the value of the 3D point X j Mapped to use camera parameters C i The projection function of the 2D image point. ij is the observed image point of the jth point in the i-th camera. ij is the weight coefficient, indicating the confidence or importance of the jth point observed by the i-th camera. C represents the set of camera parameters, including internal and external parameters; X represents the set of 3D point coordinates in the scene; n is the total number of observation points in the image; m is the total number of 3D points in the scene.
[0085] Compared with the global BundleAdjustment, the local BA adopted in the present invention pays more attention to a specific area (such as the human body area) and adopts a hierarchical or step-by-step optimization strategy to improve the computational efficiency and the accuracy of parameter optimization.
[0086] The quality of the optimization process is evaluated by calculating the optimized reprojection error using formula (3) to ensure that the error is small enough. As needed, the various steps of the algorithm may be adjusted, such as improving the stability of feature point detection, adjusting the accuracy of MASK segmentation, or adjusting the parameters of Bundle Adjustment to further improve the accuracy and quality of reconstruction.
[0087] Ultimately, this process generates sparse point clouds for the human body area, which can be used for further analysis such as human pose estimation, motion analysis, or further dense reconstruction.
[0088] Step 3: Reconstruct the dense point cloud of the human object based on the sparse point cloud of the human body area.
[0089] Based on the human body segmentation results, the human body area to be reconstructed on the image can be obtained; the depth information of the human body area can be obtained through the multi-view stereo geometry method and the locally optimized camera parameters; the sparse point cloud of the human body area is further fused with the depth information to generate a dense point cloud representing the human body object.
[0090] The step 3 specifically includes:
[0091] Step 3.1: Human body region segmentation: According to the results of human body segmentation performed by Mask R-CNN on each frame of the image, the human body region in the image is identified.
[0092] Step 3.2: Multi-view geometry and depth estimation, including triangulation and multi-view stereo matching steps.
[0093] Triangulation: For feature points that match in multiple images, the position of each point in three-dimensional space is calculated by triangulation using their pixel positions and locally optimized camera internal and external parameters to obtain depth information.
[0094] Multi-view stereo matching: By comparing the similarity of human body regions in different views, the correspondence between pixels can be estimated at a finer level to obtain the depth information of each pixel. This step can improve the accuracy of depth estimation through global optimization methods.
[0095] Step 3.3: Depth information fusion and dense point cloud reconstruction.
[0096] The depth map fusion process is to fuse the depth maps obtained from different perspectives into a unified 3D model. The depth information of multiple images is calculated, and its mathematical model can be simplified to the depth calculation formula:
[0097]
[0098] where Z represents the depth of the point relative to the camera; f is the focal length of the camera; B is the baseline distance between the two cameras in a stereo setup; and d is the difference in horizontal position of the feature in the two stereo images (disparity).
[0099] Dense point cloud generation: With depth information, a dense point cloud representing the human body can be generated. Each point in the point cloud represents a specific location on the human body surface, and the density of the point cloud is sufficient to describe the shape and characteristics of the human body in detail.
[0100] The dense point cloud of the human body object reconstructed in step 3 is obtained by further processing the sparse point cloud obtained in step 2. In step 2, the sparse point cloud is obtained by feature point extraction, matching and 3D reconstruction, which provides a preliminary 3D structure for obtaining depth information. Step 3 further obtains more detailed depth information on this basis through multi-view geometry methods and depth estimation, thereby generating a dense point cloud representing the human body. Therefore, step 3 increases the density and details of the point cloud through depth estimation and information fusion based on the sparse point cloud in step 2.
[0101] Step 4: Reconstruct a watertight human body surface mesh based on the dense point cloud of the human object.
[0102] The parameterized human body SMPLX model is used as an initial solution and a global regularization term to constrain the meshing step and reconstruct a watertight human body surface mesh. The step 4 specifically includes:
[0103] Step 4.1: Extract the SMPLX model from the image and perform an initial match with the dense point cloud of the human object.
[0104] First, the SMPLX model parameters are extracted from the captured image, and the global search and local optimization are combined to complete the process. The dense point cloud of the human object is used as a reference, so that the SMPLX model generated by these parameters is as close to the human body shape as possible. Then the SMPLX model is used as the initial value to ensure that the generated mesh can more accurately represent the human body shape.
[0105] Step 4.2: The generated mesh is made to fit the dense point cloud and maintain the geometric properties of the SMPLX model by globally minimizing the geometric error.
[0106] After the initial matching, an error function is defined to quantify the difference between the mesh to be generated and the dense point cloud. This error is usually the sum of the distances from the points in the dense point cloud to the nearest points on the surface of the generated mesh. During the meshing process, the SMPLX model is used as a global regularization term to constrain the shape of the mesh, so that the generated mesh not only matches the dense point cloud, but also maintains the geometric characteristics of the SMPLX model. By adjusting the shape and position parameters of the mesh to be generated, the gradient descent method is used to minimize the above error function while considering the regularization term of the SMPLX model. This process needs to be iterated multiple times until a predetermined convergence criterion is reached.
[0107] Step 4.3: Perform line of sight conflict detection and resolve line of sight conflicts to obtain a human body model after global optimization.
[0108] Line of sight conflict detection: During the optimization process, special attention should be paid to the problem of line of sight conflict, that is, some parts of the model mistakenly block the parts that should be visible. This can be detected by simulating the observation of the model from multiple perspectives to ensure that all lines of sight of the observation point cloud can find the corresponding visible surface on the model.
[0109] Conflict Resolution: Once a line of sight conflict is detected, it can be resolved by adjusting the pose or shape parameters of the model.
[0110] Step 4.4: Refine and seal the generated mesh to reconstruct a watertight human body surface mesh.
[0111] After the global optimization is completed, the resulting mesh parameters can generate a human body model that closely matches the shape of the dense point cloud. The model can then be further refined to ensure that the generated human body mesh is watertight, i.e., without holes. Further refinement of the model means increasing the mesh density of the model, adding more vertices and faces, in order to more accurately describe the shape and details of the human body. This includes adding more meshes in areas that require higher details, and adjusting the layout of existing meshes to better match the data in the dense point cloud.
[0112] Step 5: Optimize the human body surface mesh to restore a three-dimensional model with more human body details.
[0113] Optimize the human body surface mesh, and minimize the reprojection error of the three-dimensional model on different two-dimensional images while ensuring the consistency of the two-dimensional to three-dimensional joints and facial key points. At the same time, optimize the structural prior to restore more human details and obtain a three-dimensional model of the human body object, ensuring the surface accuracy and model compactness of the reconstructed human body object.
[0114] The step 5 specifically includes:
[0115] Step 5.1: Align 2D and 3D key points by extracting 2D key points and performing 3D reconstruction.
[0116] Extract 2D key points: Extract 2D key points of the human body and face from each captured image, which can be achieved through existing human pose estimation and facial recognition algorithms.
[0117] Establish correspondence: Determine the corresponding key point of each 2D key point in the 3D model.
[0118] 3D reconstruction: Using the principle of multi-view geometry and combining the internal and external parameters of the camera, 2D key points are reconstructed in 3D to obtain their positions in 3D space.
[0119] Step 5.2: Minimize the reprojection error.
[0120] Calculate the reprojection error: For each 3D keypoint, the camera model is used to calculate its expected position on each image. The difference between the expected position of the 2D keypoint and the actual detected position is the reprojection error.
[0121] Optimize model parameters: Adjust the parameters of the 3D model (including shape, pose, etc.) to minimize the total reprojection error of all key points.
[0122] The mesh is adjusted to fit the 3D reconstruction result as closely as possible while minimizing the geometric error and reprojection error, which can be expressed as:
[0123]
[0124] Where V and F represent the vertices and faces of the mesh respectively; P i is the 2D keypoint observed in image i; Φ is a function that depends on the camera parameters C i The 3D mesh points are projected onto the 2D image plane; Ψ is a regularization term that adds prior knowledge about the shape and posture of the human body to ensure that the mesh remains realistic; λ is a weight factor that balances the importance of data fidelity and the regularization term; k represents the number of observed 2D key points. This means that during the optimization process, the reprojection errors of all key points are considered to ensure the accuracy and consistency of the 3D model under multiple viewpoints.
[0125] Step 5.3: Application of structural priors.
[0126] Introducing structural prior knowledge: Use prior knowledge, namely the anatomical structure and motion characteristics of the human body, to guide the adjustment of the model and help restore human details in the absence of sufficient image information.
[0127] Detail recovery: Using structural priors, local details of the model, such as facial expressions or finger gestures, are refined while maintaining overall consistency and naturalness, thereby recovering a 3D model with more human details.
[0128] Step 6: Use texture mapping to visualize the 3D model and generate a 3D human body model.
[0129] According to the stereoscopic geometric correspondence, the human body color information in the two-dimensional image is mapped to the texture map and the map is globally optimized to reduce the uneven skin color problem caused by the color difference of different images; the optimized texture map is applied to the three-dimensional model, and the visual rendering is performed using graphics rendering technology to generate a three-dimensional model of the human body.
[0130] The step 6 specifically includes:
[0131] Step 6.1: Generate a texture map by extracting color information and performing texture mapping.
[0132] Extract color information: Extract the color information of the human body from each captured image, including color correction and brightness adjustment to ensure that the color information extracted from different images is consistent.
[0133] Establish correspondence: Determine the corresponding position of each image pixel on the 3D model through multi-view stereo matching, and use the internal and external parameters of the camera to calculate the 3D space point corresponding to each pixel on the image.
[0134] Generate texture map: Map the extracted color information to the surface of the 3D model to generate a texture map.
[0135] Texture coordinate mapping: Assign texture coordinates to each vertex on the model to ensure that the texture image is correctly overlaid on the 3D model.
[0136] Step 6.2: Globally optimize the map, including chromatic aberration correction, texture fusion and optimization algorithm.
[0137] Chromatic aberration correction: In order to reduce the uneven skin color caused by the color difference between different images, chromatic aberration correction is required, which is achieved through a global color correction algorithm.
[0138] Texture Fusion: In the overlapping areas of the model surface, the textures of different images may produce edges or discontinuities. Through texture fusion technology, these transition areas can be smoothed to reduce visual discontinuities.
[0139] Optimization Algorithms: The process of globally optimizing the map can be achieved by minimizing an energy function that quantifies the color differences between textures and the smoothness of transition areas. The optimization process can use gradient descent, genetic algorithms, or other optimization techniques to find the minimum value of the energy function.
[0140] Step 6.3: Apply the optimized texture map to the 3D model and use graphics rendering technology to render the final 3D human body model. This step can show a highly realistic human body model with rich details and natural and consistent colors.
[0141] The present invention proposes an automatic reconstruction method of a human body three-dimensional model based on explicit space, which takes into account the human body details and texture refinement of the reconstructed human body model, and realizes the high precision of the human body model by continuously optimizing camera parameters, reducing reprojection errors, globally minimizing geometric errors, and global mapping. Compared with the prior art, the main advantages of the present invention are: 1) The present invention forms an automatic human body three-dimensional model reconstruction method based on human body explicit space by utilizing multi-angle human body image data; 2) The geometric error can be better reduced and the camera parameters can be optimized during the modeling process; 3) The human body surface mesh that integrates the depth information of the human body region is optimized and reconstructed to ensure the surface accuracy and model compactness of the reconstructed human body; 4) A more refined human body reconstruction model can be obtained.
[0142] In some embodiments, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for automatic reconstruction of a three-dimensional human body model based on explicit space.
[0143] In some embodiments, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for automatic reconstruction of a three-dimensional human body model based on explicit space.
[0144] In some embodiments, the present invention also provides a computer device, which includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps of the method for automatically reconstructing a three-dimensional human body model based on explicit space can be implemented.
[0145] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A method for automatically reconstructing a three-dimensional human body model based on explicit space, characterized in that: include: Calibrate the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial parameters of the camera; Use a camera to shoot the human body, optimize the initial camera parameters and perform 3D reconstruction for the human body area to obtain a sparse point cloud of the human body area; Reconstructing dense point cloud of human object based on sparse point cloud of human region; Reconstruct a watertight human body surface mesh based on the dense point cloud of the human body object; Optimize the human body surface mesh to restore a 3D model with more human body details; The texture map is used to visualize the 3D model and generate a 3D human body model.
2. The method for automatically reconstructing a three-dimensional human body model based on explicit space according to claim 1, characterized in that: The method of calibrating the internal and external parameters of the camera according to the checkerboard calibration method to obtain the initial parameters of the camera specifically includes: Select a chessboard and use a camera to collect calibration images of the chessboard at different positions and angles; Use the corner detection algorithm to find the corners of the checkerboard in each calibration image and refine the positions of these corners using sub-pixel corner detection technology; Assign a three-dimensional coordinate in the world coordinate system to each corner point of the chessboard; The camera calibration algorithm is used to calculate the camera's internal and external parameters using the 2D positions of the corner points on the calibration image and the corresponding 3D coordinates. Optimize the internal and external parameters of the camera to minimize the reprojection error and obtain the optimized internal and external parameters; Use the optimized internal and external parameters to perform distortion correction and 3D reconstruction on the new calibration image to verify the accuracy of the calibration result; If the accuracy of the calibration result does not meet the requirements, recalibration is performed until the accuracy of the calibration result meets the requirements, and the optimized internal and external parameters are used as the initial parameters of the camera.
3. The method for automatic reconstruction of a human body three-dimensional model based on explicit space according to claim 2, characterized in that: The method of photographing a human body with a camera, optimizing initial camera parameters and performing three-dimensional reconstruction on a human body region to obtain a sparse point cloud of the human body region specifically includes: Use the camera to shoot human body, extract feature points and perform human body area MASK segmentation on the two-dimensional image taken each time, extract feature points in the two-dimensional image and segment the human body area; Use the feature point matching algorithm to match the feature points between different two-dimensional images, and use the matched feature points and the initial parameters of the camera to use the three-dimensional reconstruction algorithm to estimate the position of each feature point in the three-dimensional space to obtain a sparse point cloud of the human body area; Run local BundleAdjustment to optimize the initial camera parameters and the sparse point cloud coordinates of the human body area to minimize the reprojection error and obtain the locally optimized camera parameters and sparse point cloud.
4. The method for automatic reconstruction of a human body three-dimensional model based on explicit space according to claim 3, characterized in that: The method of reconstructing a dense point cloud of a human body object based on a sparse point cloud of a human body region specifically includes: The depth information of the human body area is obtained through multi-view stereo geometry method and locally optimized camera parameters; The sparse point cloud of the human body area is fused with depth information to generate a dense point cloud representing the human object.
5. The method for automatic reconstruction of a human body three-dimensional model based on explicit space according to claim 4, characterized in that: The method of reconstructing a watertight human body surface mesh based on a dense point cloud of a human body object specifically includes: Extract the SMPLX model from the captured image and perform an initial match with the dense point cloud of the human object; By globally minimizing the geometric error, the generated mesh is consistent with the dense point cloud and the geometric characteristics of the SMPLX model are maintained; Perform sight conflict detection and resolve sight conflict to obtain a human body model after global optimization; The human body model mesh is refined and sealed to reconstruct a watertight human body surface mesh.
6. The method for automatic reconstruction of a three-dimensional human body model based on explicit space according to claim 5, characterized in that: The optimization of the human body surface mesh to restore a three-dimensional model with more human body details specifically includes: Optimize the human body surface mesh, and minimize the reprojection error of the three-dimensional model on different two-dimensional images while ensuring the two-dimensional to three-dimensional consistency of the human body joints and facial key points. At the same time, optimize the structural prior to restore more human body details and obtain a three-dimensional model of the human body object.
7. The method for automatic reconstruction of a three-dimensional human body model based on explicit space according to claim 6, characterized in that: The three-dimensional model is visualized and rendered using texture mapping to generate a three-dimensional human body model, specifically comprising: Map the human body color information in the two-dimensional image to the texture map according to the stereoscopic geometric correspondence, and globally optimize the texture; The optimized texture map is applied to the three-dimensional model, and the visual rendering is performed using graphics rendering technology to generate a three-dimensional model of the human body.
8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for automatically reconstructing a three-dimensional human body model based on explicit space as described in any one of claims 1 to 7 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for automatically reconstructing a three-dimensional human body model based on explicit space as described in any one of claims 1 to 7 are implemented.
10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for automatic reconstruction of a three-dimensional human body model based on explicit space as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Rib three-dimensional reconstruction method and system based on point cloud up-sampling
CN116630531A
Three-dimensional reconstruction method, apparatus and system, medium, and computer device
WO2022237249A1
