Method and system for generating SMPL surface ground truth based on multi-view human joint constraints

Abstract

The application discloses a SMPL surface true value generation method and system based on multi-view human joint constraints, and does not need an infrared motion capture system. The steps are as follows: a multi-view camera system after calibration is built, images are synchronously collected, and two-dimensional joint nodes (including confidence) are detected; effective view angles are screened according to the confidence and effectiveness, multi-view triangulation is carried out, and abnormal view angles are iteratively removed through re-projection inspection; three-dimensional joint nodes after iteration, two-dimensional observation and a projection matrix are taken as constraints, SMPL posture, shape and global translation parameters are optimized, surface vertices, model joints and parameter true values are generated; finally, re-projection consistency, skeleton topology and coordinate uniformity verification are carried out, and the passers are taken as human three-dimensional reconstruction supervision true values. The application can stably generate spatially consistent high-quality SMPL supervision data.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D human reconstruction, sensor calibration and training data generation, and in particular to a method and system for generating ground truth values ​​of SMPL surfaces based on multi-view human joint constraints. Background Technology

[0002] Human 3D surface reconstruction tasks typically require high-quality supervised ground truth data on human pose, shape, and surface mesh. Traditional methods for obtaining supervised ground truth data often rely on optical motion capture systems, wearable inertial sensors, or manual annotation processes. While these methods can obtain high-precision motion information, they usually suffer from limitations such as restricted acquisition environments, high equipment costs, significant burden on subjects, and limited ability to expand to different scenarios.

[0003] Multi-view vision systems can acquire human images from different viewing directions and recover the three-dimensional joints of the human body using multi-view geometric constraints. Based on this, parametric human models can represent different human poses and body shapes with a unified topological human mesh, providing vertex-level supervision for the human surface reconstruction network. Therefore, generating SMPL surface ground truth based on multi-view vision systems has become an important technical approach to replace traditional motion capture schemes.

[0004] However, existing SMPL surface truth generation methods based on multi-view vision still have the following technical shortcomings: First, the quality of 2D keypoint detection results directly determines the accuracy of subsequent 3D reconstruction. However, existing methods typically rely solely on confidence thresholds for simple screening, lacking a systematic mechanism for evaluating viewpoint effectiveness. When a viewpoint experiences local occlusion or detection failure, low-quality observations still participate in triangulation calculations, leading to significant deviations in the recovered 3D keypoints.

[0005] Secondly, most existing triangulation methods are one-way open-loop processes, that is, solving for the three-dimensional coordinates once based on two-dimensional observations from each viewpoint. In practical applications, due to limitations of two-dimensional detection errors and residual errors in camera calibration, the initial triangulation results often contain outliers that deviate from the true position. Existing methods lack automatic identification and iterative correction mechanisms for such outliers, and cannot further improve the accuracy of the three-dimensional joints through closed-loop feedback.

[0006] Third, existing methods typically use only 3D joints as constraints when fitting 3D joints to the SMPL model, failing to fully utilize the rich spatial information contained in multi-view 2D observations. Relying solely on sparse joint constraints makes it difficult to drive SMPL mesh vertices to accurately fit the real human body surface, resulting in a discrepancy between the generated SMPL surface ground truth and the actual shape of the human body.

[0007] Fourth, existing methods lack a systematic quality verification framework for the generated human body surface ground truth. Due to errors in 2D keypoint detection, local occlusion, multi-camera calibration errors, and inconsistencies between different coordinate systems, the reliability of SMPL surface ground truth is reduced. Existing methods lack a mechanism for verification and screening from multiple dimensions such as reprojection consistency, skeleton topology rationality, and coordinate space consistency. SMPL fitting results of abnormal frames and frames with large errors may be directly used as supervised ground truth, thereby affecting the training effect of downstream human body 3D surface reconstruction networks.

[0008] Therefore, it is necessary to propose a surface ground truth generation method that combines multi-view camera calibration, 2D keypoint quality screening, 3D joint robust recovery, SMPL parameter optimization, and ground truth quality verification, so as to construct stable, usable, and spatially consistent supervised ground truth for human 3D surface reconstruction tasks without relying on infrared motion capture systems and wearable devices. Summary of the Invention

[0009] In view of this, the purpose of the present invention is to provide a method and system for generating SMPL surface truth values ​​based on multi-view human joint constraints. This method solves the technical problem in the prior art that the reliability of SMPL surface truth values ​​is insufficient due to two-dimensional key point detection errors, occlusion and camera calibration errors, which makes it impossible to provide stable supervised truth values ​​for human three-dimensional surface reconstruction networks.

[0010] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a method for generating the true value of SMPL surfaces based on multi-view human joint constraints, comprising the following steps: Step 1: Build a multi-view human image acquisition system containing multiple cameras; Step 2: Use the calibration board to perform intrinsic parameter calibration, distortion parameter calibration and relative extrinsic parameter calibration on the multiple cameras respectively to obtain the projection matrix corresponding to each camera; Step 3: Acquire multi-view human body images at the same time, perform two-dimensional human body key point detection on each view image to obtain two-dimensional joint point observations containing pixel coordinates and confidence scores; determine the effective observation view set for each human body joint based on the confidence score and view validity screening rules of the two-dimensional joint point observations. Step 4: Based on the set of effective observation viewpoints and the projection matrix of each camera, perform multi-view triangulation on the same human joint to obtain a three-dimensional human joint; perform reprojection verification on the initially recovered three-dimensional points. When the reprojection error of a certain viewpoint exceeds the threshold, the observation of that viewpoint is discarded and the solution is re-solved. Step 5: Using the iteratively updated 3D human joints, multi-view 2D joint observations, and camera projection matrix as constraints, establish the SMPL parameter optimization model; solve the SMPL pose parameters, shape parameters, and global translation parameters through iterative optimization to generate SMPL human surface vertices, SMPL model joints, and true values ​​of SMPL parameters. Step 6: Perform reprojection consistency verification, human skeleton topology rationality verification, and coordinate unification verification on the generated SMPL surface ground values. Use the verification results as the training supervision ground values ​​for the human 3D surface reconstruction task.

[0011] Furthermore, step 2 involves calibrating the intrinsic parameters, distortion parameters, and relative extrinsic parameters of the multiple cameras, specifically including: Step 21: Based on the checkerboard calibration board, solve the intrinsic parameter matrix of each camera by using the correspondence between the known positions of the corner points of the calibration board in physical space and their observed positions in the image. Step 22: After obtaining the corner point correspondence of each image, the initial value of the intrinsic parameter matrix is ​​first obtained by estimating the homography matrix. Then, based on this, nonlinear optimization is performed in combination with the distortion parameters to minimize the reprojection error of all corner points and solve for the radial distortion coefficient and tangential distortion coefficient. Step 23: Select one camera in the system as the reference camera. Based on the checkerboard calibration board, all cameras observe the same calibration board simultaneously. Establish cross-view correspondence using the same set of checkerboard corner points. Calculate the relative rotation matrix and translation vector from the reference camera to the m-th camera using the combination of extrinsic parameters of each camera relative to the world coordinate system. Under the condition of fixed intrinsic parameters, optimize the relative extrinsic parameters by minimizing the joint reprojection error between the reference camera and the m-th camera. Step 24: Organize the set of extrinsic parameters for the entire multi-camera system, with the reference camera as the center.

[0012] Furthermore, the specific process of solving the intrinsic parameter matrix of each camera in step 21 includes: Define the world coordinate system, camera coordinate system, image physical coordinate system, and pixel coordinate system; All corner points on the checkerboard calibration board are located on the same plane, and the third-dimensional coordinate of the corner points is always 0. A one-to-one correspondence is established between the known three-dimensional coordinates of the checkerboard corner points in physical space and their two-dimensional observation coordinates in the image. Based on the projection model, the projection relationship from the 3D coordinates of the checkerboard corner points to the pixel coordinates is simplified to the homography relationship from the plane to the image:

[0013] Where s is the scale factor; Represents pixel coordinates; This represents the horizontal coordinate of the corner point of the chessboard grid in the calibration plate coordinate system (world coordinate system); This represents the vertical coordinate of the corner point of the chessboard grid in the calibration plate coordinate system (world coordinate system); H is the homography matrix from the calibration plate plane to the image plane; Different homography matrices correspond to different poses of the checkerboard image. Multiple homography matrices together impose constraints on the camera intrinsic parameters, thereby achieving intrinsic parameter estimation.

[0014] Furthermore, the objective function for minimizing the reprojection error of all corner points in step 22 is:

[0015] in, This represents the actual detected corner pixel coordinates. This indicates the reprojection point estimated from the projection model based on the current parameters. and This represents the extrinsic parameter corresponding to the q-th calibration image; Q represents the number of calibration image groups, m represents the number of rows of the checkerboard corner points, K is the camera intrinsic parameter matrix, and D is the distortion parameter vector; i represents the row index of a corner point on the chessboard; j represents the column index of the corner point of the chessboard.

[0016] Furthermore, in step 23, when optimizing the relative extrinsic parameters, the objective function for the joint reprojection error between the reference camera and the m-th camera is:

[0017] in, Let 'not' represent the actual corner pixel coordinates detected in the reference camera and the m-th camera. This indicates the corner coordinates obtained by projecting based on the current parameters; Indicates the distance from the reference camera coordinate system to The coordinate system of the m-th camera The relative rotation matrix; Indicates the distance from the reference camera coordinate system to The coordinate system of the m-th camera The relative translation vector; Indicates the index of the calibration image group.

[0018] Furthermore, step 3 involves performing two-dimensional human keypoint detection on images from various viewpoints to obtain two-dimensional keypoint observations containing pixel coordinates and confidence levels, specifically including: Two-dimensional human keypoint detection is performed on the synchronized multi-view images. A convolutional neural network is then used to jointly predict the keypoint heatmap and limb association field, outputting a set of two-dimensional human keypoints.

[0019] in, This represents the pixel coordinates of the j-th joint in the image plane. This represents the detection confidence level of the corresponding key point, where J is the total number of human key points; Let v represent the set of two-dimensional key points of the human body detected from the v-th viewpoint at time t; Preprocessing of 2D key point observations: remove key point observations with confidence levels below the preset threshold τ, and screen and filter abnormal detection results in combination with human body topology. When false detections or missed detections occur in individual perspectives, only valid perspectives are retained for subsequent triangulation.

[0020] Furthermore, the objective function of the SMPL parameter optimization model established in step 5 is:

[0021] in, This represents the error term between the model joints projected onto the multi-view image plane and the 2D keypoint detection results. This represents the error term between the model joint and the true value of the 3D joint. This represents the prior constraints on human posture. Regularization terms representing shape parameters and other model parameters. , , , Assign weights to each loss term; The three-dimensional joint constraints are:

[0022] In the formula, This represents the position of the j-th model joint generated by the current SMPL parameters. This represents the ground truth value of the j-th 3D joint obtained by multi-view triangulation. The two-dimensional reprojection constraint terms are:

[0023] in, Let v be the projection matrix of the v-th camera. This represents the two-dimensional observation position of the j-th joint from the v-th viewpoint. For the corresponding confidence weights, This represents the process of homogeneous projection onto pixel coordinates.

[0024] Furthermore, step 6 involves verifying the coordinate uniformity of the generated SMPL surface truth value, specifically including: Based on the cross-sensor extrinsic calibration results, the SMPL joints and vertices are transformed from the camera coordinate system to the lidar coordinate system:

[0025] in, and These represent rotation and translation transformations from the camera coordinate system to the lidar coordinate system, respectively. Where J_C and V_C represent the joint and vertex coordinates of the SMPL model in the camera coordinate system, respectively; J_L and V_L represent the joint and vertex coordinates of the SMPL model in the lidar coordinate system, respectively; and R_CL and t_CL represent the rotation matrix and translation vector from the camera coordinate system to the lidar coordinate system, respectively.

[0026] The present invention provides a SMPL surface truth generation system based on multi-view human joint constraints, comprising: The multi-view image acquisition module is used to build a multi-view human body image acquisition system containing multiple cameras to acquire multi-view human body images at the same time. The camera calibration module is used to perform intrinsic parameter calibration, distortion parameter calibration and relative extrinsic parameter calibration on the multiple cameras using a calibration board, so as to obtain the projection matrix corresponding to each camera; The two-dimensional joint detection and effective viewpoint selection module is used to perform two-dimensional human keypoint detection on images from various viewpoints, obtain two-dimensional joint observations containing pixel coordinates and confidence scores, and determine the effective observation viewpoint set for each human joint based on the confidence scores and viewpoint effectiveness selection rules of the two-dimensional joint observations. The three-dimensional joint iterative recovery module is used to perform multi-view triangulation on the same human joint based on the effective observation view set and the projection matrix of each camera to obtain the three-dimensional human joint, and to perform reprojection verification on the initially recovered three-dimensional points. When the reprojection error of a certain view exceeds the threshold, the observation of that view is discarded and the solution is re-solved. The SMPL parameter joint optimization module is used to establish an SMPL parameter optimization model with the iteratively updated 3D human joints, multi-view 2D joint point observations and camera projection matrix as constraints. Iterative optimization solves the SMPL pose parameters, shape parameters and global translation parameters, and generates SMPL human surface vertices, SMPL model joints and SMPL parameter ground values. The triple verification and supervised ground truth generation module is used to perform reprojection consistency verification, human skeleton topological rationality verification, and coordinate unification verification on the generated SMPL surface ground truth. The verification results are used as the training supervised ground truth for the human 3D surface reconstruction task.

[0027] Furthermore, the camera calibration module includes: The intrinsic parameter calibration submodule is used to solve the intrinsic parameter matrix and distortion parameters of each camera based on the chessboard calibration board and the correspondence between the known positions of the corner points of the calibration board in physical space and their observed positions in the image. The extrinsic parameter calibration submodule is used to select one camera in the system as a reference camera. Based on the checkerboard calibration board, it uses the same set of checkerboard corner points when each camera observes the same calibration board at the same time to calculate the relative rotation matrix and translation vector from the reference camera to the m-th camera. The calibration and verification submodule is used to calculate the average reprojection error on all calibration images of a single camera, the average joint reprojection error between the reference camera and the m-th camera, and to draw the actual detected corner points and reprojected corner points in the calibration images for visual comparison. Based on the external parameter calibration results, the relative positions of each camera are restored and the spatial structure consistency is verified with the actual deployment scene.

[0028] The beneficial effects of this invention are as follows: This invention provides a method for generating SMPL surface ground truth values ​​based on multi-view human joint constraints. The method includes: constructing a multi-view camera acquisition system and calibrating the camera; acquiring multi-view human images at the same time and performing two-dimensional joint point detection to obtain two-dimensional joint point observations containing pixel coordinates and confidence scores; determining a set of effective observation viewpoints based on confidence scores and viewpoint validity filtering rules; performing multi-view triangulation based on the effective observation viewpoint set and camera projection matrix, reprojecting and iteratively eliminating abnormal viewpoints from the initially recovered three-dimensional points; establishing an SMPL parameter optimization model with the iteratively updated three-dimensional human joints, two-dimensional joint point observations, and camera projection matrix as constraints, iteratively solving for posture parameters, shape parameters, and global translation parameters to generate SMPL surface vertices, model joints, and parameter ground truth values; verifying the generated SMPL surface ground truth values ​​through reprojection consistency, human skeleton topology rationality, and coordinate unification, and using the verified results as training supervision ground truth values ​​for the human three-dimensional surface reconstruction task. This invention can generate stable, usable, and spatially consistent SMPL surface supervision ground truth values ​​without relying on an infrared motion capture system. Compared with existing technologies, this invention has the following beneficial effects: First, this invention determines the set of valid observation perspectives by combining confidence level and perspective validity screening rules, effectively eliminating low-quality 2D joint point observations caused by local occlusion, motion blur, or detection failure. Compared to the existing technology that only uses confidence level for weighted fusion, the perspective screening mechanism of this invention can filter unreliable perspectives at the source, reducing the contamination of subsequent triangulation reconstruction by erroneous observations and significantly improving the robustness and accuracy of 3D human joint restoration. Second, this invention constructs a closed-loop optimization mechanism of "triangulation-verification-correction-retriangulation" by reprojecting and iteratively eliminating abnormal perspectives on the initially restored 3D points. This mechanism can adaptively identify and eliminate abnormal perspectives with reprojection errors exceeding the threshold. Compared to the open-loop processing method of single triangulation in the existing technology, it effectively suppresses 3D point drift caused by 2D detection errors or residual camera calibration errors, making the restored 3D human joints more accurate and stable in spatial position. Third, this invention establishes an SMPL parameter optimization model using iteratively updated 3D human joints, multi-view 2D joint observations, and camera projection matrices as joint constraints, achieving integrated iterative solution of SMPL pose parameters, shape parameters, and global translation parameters. Unlike existing fitting methods that rely solely on 3D joints or simply on 2D projection constraints, the triple geometric constraints of this invention complement and correct each other, ensuring that the generated SMPL human mesh maintains a high degree of consistency with the real human skeleton in 3D space, while remaining aligned with image observations in multi-view projections, significantly improving the accuracy and spatial consistency of the SMPL surface ground truth. Fourth, this invention forms a complete multi-dimensional quality assurance system by performing reprojection consistency verification, human skeleton topology rationality verification, and coordinate unification verification on the generated SMPL surface ground truth. Reprojection consistency verification ensures geometric consistency between the SMPL mesh and multi-view image observations; skeleton topology rationality verification ensures that the human skeleton conforms to anatomical structural constraints, avoiding abnormal bone segment lengths or skeletal distortion; coordinate unification verification ensures that the SMPL ground truth is consistent with the coordinate system of downstream training data. The triple verification is independent yet complementary, removing abnormal frames identified from the SMPL optimization results to ensure the reliability and usability of the final output training supervision ground truth. Fifth, this invention uses the verified results as the training supervision ground truth for the human 3D surface reconstruction network, realizing a complete generation chain from multi-view images to SMPL surface ground truth. Compared to existing technologies that only output 3D joints or the 3D reconstruction model itself, the SMPL surface vertex-level ground truth generated by this invention can be directly used for supervised training of the human 3D surface reconstruction network, providing a data foundation for high-quality human reconstruction without relying on infrared motion capture systems, and has significant application value and promising prospects.

[0029] The above and other objects, advantages, and features of the present invention will be more fully set forth and demonstrated through the following detailed description of specific embodiments in conjunction with the accompanying drawings. Those skilled in the art, upon referring to the following detailed description and the accompanying drawings, will be able to better understand and realize the above advantages of the present invention. Other objects, features, and advantages of the present invention will become clearer after being described in detail in the detailed description section in conjunction with the accompanying drawings. Attached Figure Description

[0030] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following drawings are provided for illustration.

[0031] Figure 1 The flowchart shows the SMPL surface truth generation method based on multi-view human joint constraints. Figure 2 This is a map showing the camera positions in the experimental scene. Figure 3 This is a schematic diagram of a camera imaging model; Figure 4 This is a schematic diagram illustrating the relative positions of the four cameras. Figure 5 A schematic diagram of the projection error of the four-camera intrinsic parameter calibration. Figure 6 A schematic diagram of the projection error of the four-camera intrinsic parameter calibration. Figure 7 A schematic diagram illustrating the projection error of the four-camera external parameter calibration. Figure 8 A schematic diagram illustrating the projection error of the four-camera external parameter calibration. Figure 9 Schematic diagram of multi-view 2D human key point detection results; Figure 10 Schematic diagram of the three-dimensional human skeleton reconstruction results; Figure 11 This is a schematic diagram of the verification results of 3D joint reprojection; Figure 12 This is a graph used to verify the consistency of SMPL results through multi-view projection. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0033] Example 1

[0034] like Figure 1 As shown, Figure 1This is a flowchart of the SMPL surface truth generation method based on multi-view human joint constraints. The SMPL surface truth generation method based on multi-view human joint constraints provided in this embodiment specifically includes the following steps: S1, as Figure 2 As shown, a multi-view human body image acquisition system with multiple cameras was built.

[0035] The multi-view camera module is used to simultaneously acquire human image information from different observation directions. Assuming the system has M cameras deployed in total, the image acquired by the m-th camera at time t can be represented as: (1-1) Where H, W, and C represent the image height, width, and number of channels, respectively. Therefore, the set of multi-view images corresponding to time t can be uniformly written as: (1-2) If we consider image observations at a single moment, then the basic observation set of the radar-visual collaborative sensing system at time t is: (1-3) in, This represents the raw multimodal observation results collected by the system at time t.

[0036] S2, such as Figure 4 As shown, the multi-view intrinsic and extrinsic parameter calibration model is constructed as follows: In the multi-view human 3D surface reconstruction task, the main role of the multi-camera system is to acquire human image information from different observation directions and provide multi-view geometric constraints for subsequent 3D keypoint restoration and supervision information generation. To establish the spatial correspondence between images from different perspectives, camera calibration must first be completed, i.e., solving for the intrinsic parameter matrix, distortion parameters, and relative pose relationships between different cameras. For a multi-view system, the essence of calibration is to establish a unified projection model from 3D spatial points to the image planes of each camera.

[0037] First, clarify the relevant definitions of each coordinate system in the model: (1) World coordinate system: denoted as {W}, used to describe the true three-dimensional position of the target point in the scene. During calibration, the plane containing the calibration plate or a pre-defined reference system is usually used as the world coordinate system. Let the coordinates of spatial point P in the world coordinate system be: (1-4) (2) Camera coordinate system: denoted as {C}, with its origin located at the camera's optical center. The axis usually points forward along the optical axis. axis The axes are parallel to the horizontal and vertical directions of the imaging plane, respectively. The coordinates of spatial point P in the camera coordinate system are: (1-5) (3) Image physical coordinate system (imaging plane coordinate system): denoted as {I}, its origin is usually taken near the principal point, and the unit is generally millimeters or other physical length units. The physical coordinates of point P in the imaging plane are: (1-6) (4) Pixel coordinate system: denoted as {P}, its origin is usually located at the upper left corner of the digital image, and the unit is pixels. The coordinates of point P in the digital image are: (1-7) The essence of intrinsic and extrinsic parameter calibration is to decipher the mapping relationship from the world coordinate system to the pixel coordinate system, and then model the mapping relationship in stages: The mapping relationship between the world coordinate system and the camera coordinate system is determined by the camera's extrinsic parameters. Due to differences in the installation positions and attitudes of different sensors in space, and their respective adherence to independent local coordinate system descriptions, the coordinate representation of the same target in different sensor coordinate systems is usually inconsistent. However, the prerequisite for multi-sensor collaborative sensing is the construction of a unified spatial reference frame, transforming the data collected by each sensor to the same coordinate system for expression and correlation. Only by achieving cross-sensor coordinate system unification can a consistent description of the target's spatial position be achieved, laying the foundation for subsequent data fusion, target localization, 3D reconstruction, and the development of related algorithms.

[0038] Sensor coordinate system transformation can be achieved through translation and rotation.

[0039] Assuming the camera coordinate system is the world coordinate system rotated alpha degrees around the X-axis, then: (1-8) The matrix form is: (1-9) Assuming it rotates about the Y-axis degrees, assuming it rotates about the Z-axis Multiplying the rotation components in the three directions yields the rotation matrix for converting the world coordinate system to the camera coordinate system: (1-10) Transforming from the world coordinate system to the camera coordinate system requires adding translation components in three directions. The translation matrix is: (1-11) Then spatial points in the world coordinate system The expression in the camera coordinate system is as follows: (1-12) Using homogeneous coordinates, it can be written as: (1-13) This transformation illustrates that the same camera will have different camera coordinate representations at different spatial points, and this difference is the source of multi-view geometric constraints.

[0040] Once spatial points are represented in the image's physical coordinate system, the classic camera imaging process can be described by the pinhole imaging model. Based on the aforementioned perspective projection relationship, we have: (1-14) Where f represents the focal length.

[0041] The aforementioned projection relationship is based on the ideal pinhole imaging model, assuming that the imaging system has no lens distortion. However, in actual imaging, to improve image quality and field of view, cameras are typically equipped with optical lens groups. The introduction of lenses causes the light propagation path to deviate from the ideal pinhole model, resulting in a deviation between the actual imaging result and the ideal projection relationship. This is mainly due to two factors: first, the curved structure of the lens itself alters the refraction path of light; second, due to manufacturing and assembly errors, the optical axis of the lens, the imaging plane, and the ideal optical center often cannot maintain a perfectly ideal geometric relationship, further causing image point position shifts.

[0042] Imaging deviations caused by the optical properties of a lens are generally called radial distortion. In an ideal pinhole camera model, a straight line in space remains a straight line on the image plane after projection. However, under actual imaging conditions, due to lens distortion, straight lines in a real scene often appear as curved curves in the image, and this phenomenon is more pronounced when far from the principal point and near the image edges. Radial distortion can generally be divided into two categories: barrel distortion and pincushion distortion. Barrel distortion is characterized by a decrease in image magnification as the distance from the optical axis increases; pincushion distortion, on the other hand, is characterized by an increase in image magnification as the distance from the optical axis increases.

[0043] The distortion caused by the inability to ensure that the lens and the imaging plane are perfectly parallel during camera assembly is called tangential distortion. Will Figure 3 In the imaging plane Point coordinates are written in polar coordinate form, where r represents The distance of a point from the origin of the coordinate system, θ represents the angle between the point and the horizontal axis.

[0044] For radial distortion, the coordinate changes before and after distortion can be described by a polynomial function: (1-15) For tangential distortion, we have: (1-16) Together, we obtain a result described by five distortion parameters. The final correct position of the point is: (1-17) in, The radial distortion coefficient is mainly used to describe barrel or pincushion distortion at the edges of an image. denoted as the tangential distortion coefficient.

[0045] However, in a camera, what is ultimately obtained are individual pixels, which need to be sampled and quantized on the imaging plane. To describe the process by which the sensor converts the sensed light into image pixels, a pixel plane is fixed on the physical imaging plane. It was obtained on the pixel plane. .

[0046] (1-18) in, Represents pixel coordinates, Principal point coordinates Let represent the equivalent focal lengths of the image horizontally and vertically, respectively. If distortion is ignored, then... , The above formula then degenerates into the pixel coordinate expression under the ideal pinhole imaging model.

[0047] Furthermore, the camera intrinsic parameter matrix can be represented as: (1-19) Rewriting equation (1-18) in homogeneous coordinates, we get: (1-20) Multiplying equation (1-13) by the intrinsic parameter matrix on the left yields the projection relationship from the world coordinate system to the pixel coordinate system as follows: (1-21) Where s is the scale factor, which is usually equal to the depth of the point in the camera coordinate system. This formula represents the most fundamental projection model in multi-view camera calibration and 3D reconstruction. Therefore, the core of the calibration problem lies in solving for the camera's intrinsic parameter matrix K and extrinsic parameters. This establishes a mapping relationship between real-world coordinates and pixel coordinates.

[0048] S21. Multi-camera Intrinsic Parameter Calibration Method Based on Checkerboard Calibration Board: After completing the description of the multi-camera calibration problem and the definition of the coordinate system, it is necessary to further solve for the intrinsic parameter matrix and distortion parameters of each camera. Camera intrinsic parameters determine the projection scale and principal point position of points in 3D space on the image plane, while distortion parameters describe the deviation of the real lens imaging relative to the ideal pinhole model. Therefore, the two together constitute the core parameters of a single camera imaging model. Since the checkerboard calibration board has advantages such as regular corner point distribution, clear planar structure, ease of detection, and convenient implementation, a planar calibration method based on the checkerboard calibration board is adopted to calibrate the intrinsic parameters of multi-view cameras. This method is based on the correspondence between the known positions of the calibration board corner points in physical space and their observed positions in the image. Combined with multiple calibration images under different poses, the intrinsic parameter matrix and distortion parameters of each camera are solved.

[0049] Let M and N be the number of rows and columns of the interior corner points of the checkerboard calibration plate, respectively, and let d be the physical side length of a single square. Since the calibration plate is a planar structure, a planar reference coordinate system can be established on its surface, and the world coordinates of all corner points can be uniformly represented as: (1-22) in, This represents the three-dimensional coordinates of the corner point in the i-th row and j-th column within the calibration board coordinate system (which can also be used as the world coordinate system). Since all corner points lie on the same plane, their third-dimensional coordinate is always 0. This expression demonstrates that, given the dimensions of the chessboard and the arrangement of the corner points, the precise physical position of each corner point on the calibration board can be obtained.

[0050] For a calibration image acquired by the m-th camera, the corresponding corner pixel coordinates can be represented as: (1-23) in, This represents the pixel coordinates of the corner point in the m-th camera image. Thus, a clear one-to-one correspondence is established between the known three-dimensional coordinates of the checkerboard corner point in physical space and its two-dimensional observed coordinates in the image, which is the basis for solving the camera intrinsic parameters.

[0051] According to equation (1-21), for the corner points on the chessboard marking board, since Therefore, the projection model can be simplified to: (1-24) in, and Let R be the first two column vectors of the rotation matrix R. This can be further expressed as a homography relation from a plane to an image: (1-25) Here, H is the homography matrix from the calibration plate plane to the image plane. Therefore, under the condition of a planar calibration plate, the problem of solving the camera intrinsic parameters can be transformed into a parameter solving problem under the constraints of multiple planar homography matrices. The checkerboard image under different poses will correspond to different homography matrices, and these homography matrices collectively impose constraints on the camera intrinsic parameters, thereby enabling intrinsic parameter estimation.

[0052] However, in actual camera imaging processes, lens distortion often occurs, thus requiring further calculation of distortion parameters. According to equation (1-17), the intrinsic parameter calibration results for the m-th camera can be uniformly expressed as: (1-26) in, Let represent the distortion parameter vector of the m-th camera. Therefore, the complete intrinsic parameter model of a single camera can be described by the intrinsic parameter matrix and the distortion parameters.

[0053] Next, we proceed to the solution step. In the actual implementation, intrinsic parameter calibration first requires detecting the checkerboard corner points from multiple calibration images. Let the m-th camera acquire Q checkerboard images at different poses; then each image corresponds to a set of corner point observations: (1-27) Correspondingly, the set of physical corner points on the chessboard plane is denoted as: (1-28) Therefore, the problem of solving camera intrinsic parameters can be regarded as using multiple sets of... The problem of estimating camera parameters based on the 2D to 3D correspondence. Since the accuracy of corner detection directly affects the calibration accuracy, sub-pixel-level optimization is usually required based on the initial corner detection to improve the accuracy of corner localization.

[0054] After obtaining the corner point correspondences of each image, the initial values ​​of the intrinsic parameter matrix can be obtained by estimating the homography matrix. Then, based on this, nonlinear optimization is performed using distortion parameters to minimize the reprojection error of all corner points. For the m-th camera, the optimization objective function can be expressed as: (1-29) in, This represents the actual detected corner pixel coordinates. This indicates the reprojection point estimated from the projection model based on the current parameters. and Let q represent the extrinsic parameters corresponding to the q-th calibration image. This optimization problem essentially involves minimizing the projection error to estimate the optimal camera intrinsic and distortion parameters, which is a standard practice in actual camera calibration.

[0055] The multi-view camera system employed allows each camera to independently complete the aforementioned intrinsic parameter calibration process. Therefore, in a multi-camera scenario, the system's intrinsic parameter results can be organized as follows: (1-30) S22. Multi-camera extrinsic parameter calibration method based on checkerboard calibration board: After solving for the intrinsic parameter matrices and distortion parameters of each camera, it is necessary to further determine the relative pose relationships between different cameras in the multi-camera system, i.e., multi-camera extrinsic parameters. Camera extrinsic parameters describe the spatial position and orientation of a camera coordinate system relative to a reference coordinate system, and are the basis for establishing multi-view geometric constraints and carrying out 3D reconstruction. For the task of this system, subsequent multi-view triangulation to recover 3D joints depends on accurate multi-camera relative poses; therefore, multi-camera extrinsic parameter calibration is a key step in the camera calibration process.

[0056] From a geometric perspective, if a certain spatial point lies in the reference camera coordinate system The coordinates below are: (1-31) Then the point is in the m-th camera coordinate system The coordinates below can be represented as: (1-32) in, The rotation matrix from the reference camera to m cameras, Let be the corresponding translation vector. If homogeneous coordinates are used, it can be written as: (1-33) Therefore, the goal of multi-camera extrinsic calibration is to solve the rigid body transformation matrix of each camera relative to the reference camera. ; For a multi-view system containing M cameras, this system selects one camera as the reference camera, usually denoted as . Then other cameras The extrinsic parameters are all defined relative to the reference camera. The advantage of doing this is that it allows the geometric relationships of a multi-camera system to be organized with a unified reference frame, which facilitates subsequent calculations of the projection and triangulation of 3D points across different viewpoints.

[0057] Let the extrinsic parameters of the m-th camera in the system be... Then the extrinsic parameters of the entire multi-camera system can be uniformly expressed as: (1-34) The reference camera itself satisfies the following: (1-35) Based on the above definition, the relative relationships between any cameras in the system can be uniformly expressed using a reference camera.

[0058] In this system, multi-camera extrinsic calibration is still performed using a checkerboard calibration board. Assume that at the same time, all cameras simultaneously observe the checkerboard calibration board, and all corresponding corner points can be detected. Since the intrinsic parameters of each camera were obtained in the previous section, the current problem can be viewed as estimating the relative pose relationships between cameras using the same set of checkerboard corner points observed by each camera, given the intrinsic parameters.

[0059] For the i-th and j-th corner points of the chessboard grid in the m-th camera, their projection relationship can be expressed as: (1-36) in Let these be the coordinates of the corner point in the world coordinate system. Let it be the pixel coordinates in the m-th camera image. The camera intrinsic parameter matrix is ​​known.

[0060] If the reference camera and the m-th camera By establishing the aforementioned projection relationships, the relative rotation and translation between the two cameras can be estimated using the corner points of the same chessboard grid corresponding to the two cameras. Furthermore, the relative extrinsic parameters between the reference camera and the m cameras can be obtained by combining their respective extrinsic parameters relative to the world coordinate system. (1-37) This formula shows that as long as the observation relationship of each camera to the same calibration plate can be obtained, the relative pose between the cameras can be further established.

[0061] Let the sets of corner points detected by the reference camera and the m-th camera in the q-th calibration image be . (1-38) The corresponding set of physical corner points of the chessboard is still denoted as g. In a fixed... , , , Under these conditions, the relative external parameters can be optimized. To minimize the reprojection error between the two cameras, the objective function can be expressed as: (1-39) in, Let 'not' represent the actual corner pixel coordinates detected in the reference camera and the m-th camera. This represents the corner coordinates obtained by projecting based on the current parameters. Essentially, this optimization process utilizes multiple sets of corresponding corner points on a chessboard grid to minimize the projection error between the two cameras under a unified geometric relationship, thereby estimating the optimal relative pose.

[0062] In this system, the final results of multi-camera extrinsic calibration are uniformly organized around the reference camera. For each non-reference camera in the system, the extrinsic parameter result of the m-th camera can be expressed as: (1-40) Therefore, by combining the external parameters of the entire system with the aforementioned intrinsic parameter results, a complete set of calibration parameters for the multi-camera system can be formed: (1-41) This parameter set will be further used in subsequent multi-view triangulation, LiDAR to camera extrinsic calibration, and training label generation processes.

[0063] S23. Accuracy Evaluation and Spatial Consistency Verification of Intrinsic and Extrinsic Parameter Calibration Results: After solving the intrinsic and extrinsic parameters of the multi-camera system, it is necessary to verify the accuracy and stability of the calibration results. This is because the camera calibration results directly determine the geometric accuracy of the subsequent multi-view triangulation reconstruction of 3D joints. If there are large errors in the calibration parameters, it will lead to deviations in the reprojection position of the same spatial point under different viewpoints, thus affecting the reliability of the 3D reconstruction results. Therefore, quantitative error analysis and qualitative visualization verification of the calibration results are indispensable steps in the multi-camera calibration process.

[0064] From the perspective of verification objectives, this system mainly analyzes the multi-camera calibration results from two aspects: firstly, verifying the reliability of the intrinsic parameters and distortion parameters of a single camera; secondly, verifying whether the solutions for the relative extrinsic parameters of the multiple cameras can correctly describe the geometric relationships between different cameras. To this end, this system uses reprojection error as the main quantitative evaluation index, and combines corner reprojection visualization to assist in the analysis of the calibration results.

[0065] (1) Analysis of reprojection error of single-camera intrinsic parameter calibration results For the m-th camera, let the actual detected pixel coordinates of the i-th and j-th checkerboard corner points in the q-th calibration image be: (1-42) The reprojection point coordinates calculated using the obtained intrinsic parameter matrix, distortion parameters, and corresponding image pose parameters are: (1-43) The reprojection error of this corner point can then be defined as: (1-44) Furthermore, the average reprojection error of the m-th camera on all calibration images can be expressed as: (1-45) Here, Q represents the number of image sets used for calibration, and M and N represent the number of rows and columns of the checkerboard corner points, respectively. This metric measures how well a single camera's imaging model fits the actual corner point observations. Generally, the smaller the average reprojection error, the more reliable the camera's intrinsic parameter calibration results.

[0066] (2) Analysis of reprojection error of multi-camera extrinsic parameter calibration results In a multi-camera system, verifying the intrinsic parameter accuracy of a single camera is insufficient; further verification of the relative poses between cameras is also necessary. Let the reference camera be... The m-th camera is For the same set of chessboard corner points, using the intrinsic and extrinsic parameters of the reference camera and the m-th camera, the reprojection position of the corner point under the two viewpoints can be calculated respectively. Therefore, the joint reprojection error of the reference camera and the m-th camera can be defined as... (1-46) By further averaging across all calibration images, the average joint reprojection error between the reference camera and the m-th camera can be obtained: (1-47) This indicator can simultaneously reflect the projection consistency of two cameras under unified extrinsic constraints, and therefore can serve as an important measure of the accuracy of multi-camera extrinsic calibration. A smaller value indicates that the relative rotation and translation relationship between the reference camera and the m-th camera can better describe the true geometric constraints between the two viewpoints.

[0067] (3) Visual verification and calibration accuracy evaluation of corner point reprojection While quantitative error can provide overall calibration accuracy, it cannot intuitively reflect the spatial distribution of errors in the image. Therefore, this system further employs corner point reprojection visualization to perform qualitative analysis of the calibration results. Specifically, the actual detected checkerboard corner points and the corner point positions obtained by reprojection using calibration parameters are simultaneously plotted in the calibration image, and the degree of overlap between the two is observed.

[0068] To verify the reliability of the multi-view camera intrinsic parameter calibration results, this system collected 14 sets of checkerboard calibration images for parameter solving and error analysis. Figure 4 The reprojection error of the four camera intrinsic parameter calibrations is visualized. Figure 5The images show close-up details of the corresponding error results. Tables 1 and 2 summarize the reprojection errors of each camera under different calibration image sets. Through joint analysis of quantitative and visual results, the calibration accuracy of the intrinsic parameters of each camera can be comprehensively evaluated. Red represents the reprojection results, and green represents the corner points automatically identified by the algorithm. Visually, the reprojected corner points largely coincide with the actual detected corner points, with only minor deviations in certain areas, indicating good intrinsic parameter calculation performance.

[0069] Table 1 Internal Parameter Calibration Weight Projection Error Table 1

[0070] Table 2 Internal Parameter Calibration Weight Projection Error Table 2

[0071] Overall, the reprojection errors of the intrinsic parameter calibration for all four cameras reached sub-pixel level, indicating that the calibration method used in this system can accurately estimate the intrinsic parameter matrix and distortion parameters of each camera, resulting in good overall calibration performance. The sub-pixel level reprojection error demonstrates that when the corner points of the chessboard are back-projected using the obtained camera parameters, the calculated corner point positions show a high degree of consistency with the actual detected positions, thus verifying the effectiveness of the single-camera imaging model fitting results.

[0072] Further analysis of the error distribution of each camera under different calibration image sets revealed that camera 1 exhibited a maximum reprojection error of 3.514 pixels in the last set of data. Inspection of this set of images showed that the main cause was a single corner point localization error during the checkerboard corner point detection process, leading to a significant increase in the local error of this set of data. However, the overall statistical results of the 14 sets of data showed that the average reprojection error of camera 1 remained at a low level, indicating that this local anomaly did not significantly affect the overall intrinsic parameter calibration results. This demonstrates that under the joint constraint of multiple sets of high-quality calibration images, the local corner point detection error in individual images can be mitigated to some extent during the overall optimization process, thus ensuring that the calibration results still have good stability.

[0073] To verify the reliability of the multi-view camera extrinsic parameter calibration results, this system uses 14 sets of checkerboard calibration images to perform error analysis on the multi-camera extrinsic parameters. Figure 6 The reprojection error visualization results of multi-camera extrinsic calibration are presented. Figure 7 This is a close-up of the corresponding result. Figure 8 for Figure 7 A close-up view of a specific area is used to magnify the joint reprojection error of external parameter calibration, allowing for a clearer display of the error distribution details. From Figure 8It can be seen that the deviation between the reprojected corner point and the actual detected corner point in the local area further verifies the effectiveness of the external parameter calibration results.

[0074] Table 3 External Parameter Weighting Projection Error Table 1

[0075] Table 4 External Parameter Weighting Projection Error Table 2

[0076] Tables 3 and 4 respectively present the joint reprojection error of different camera pairs under each set of calibration images. It can be seen that although the reprojected corner point and the actual detected corner point basically still point to the same corner point, this indicates that the extrinsic parameter calculation results are still correct.

[0077] Overall, the joint reprojection error of the extrinsic parameter calibration for each camera pair is relatively small, with most errors remaining below 3 pixels. This indicates that the relative rotation and translation parameters of the multi-camera system obtained in this study can effectively describe the spatial geometric relationships between different cameras. The small joint reprojection error shows that, under unified extrinsic parameter constraints, the same set of checkerboard corner points observed by the reference camera and the target camera have good consistency, thus verifying the effectiveness of the multi-camera extrinsic parameter calibration results. Further analysis of the error distribution of each set of data reveals that camera 1 exhibits the largest joint reprojection error of 7.041 pixels in the 7th set of data. Examination of this set of images revealed that the main reason for this is the extreme pose of the checkerboard, which leads to a significantly higher error in this set compared to other sets.

[0078] However, the overall statistical results of all 14 sets of data show that the average joint reprojection error of the camera pair is still below 3 pixels, indicating that the local anomaly did not significantly affect the overall extrinsic calibration results. Under the joint constraint of multiple sets of high-quality calibration images, the local errors in individual images can be weakened to a certain extent during the overall optimization process, thus ensuring that the overall extrinsic results of the multi-camera system still have good stability.

[0079] In addition to joint reprojection error analysis, this system further verifies the multi-camera extrinsic parameter calibration results from the perspective of spatial structure consistency. Figure 4 The relative position map of the four cameras, recovered based on the external parameter calibration results, is presented. Figure 2 The experiment shows the actual deployment of the four cameras in the experimental field, reflecting the installation position and orientation of each camera in the scene. The comparison between the two figures shows that the extrinsic parameter recovery results are in good consistency with the actual scene deployment in terms of the relative position, orientation distribution and overall layout structure of the cameras, indicating that the obtained extrinsic parameters can realistically describe the spatial geometric relationship of the multi-camera system.

[0080] In summary, based on the joint reprojection error statistics and the visual comparison of camera relative positions, it can be concluded that the multi-camera extrinsic parameters obtained by this system can accurately describe the spatial geometric relationship between the four cameras, meeting the accuracy requirements for subsequent multi-view triangulation recovery and cross-view data association.

[0081] S3, A True-Value Restoration Method for 3D Human Joints Based on Multi-View Constraints To construct supervised information suitable for human 3D surface reconstruction, it is first necessary to recover the skeletal structure of the human body in 3D space from synchronously acquired multi-view images. Multi-view images provide richer texture and contour information for human joint localization, offering stable pose priors for subsequent parametric human model fitting. Therefore, this system, based on synchronous images from multi-view cameras, first uses OpenPose to detect 2D keypoints of the human body from each viewpoint. Then, combining the multi-camera calibration parameters obtained in Chapter 2, it uses multi-view geometric constraints to triangulate and recover the human joints, obtaining the true 3D human joint values. Finally, the effectiveness of the recovery results is evaluated through reprojection error analysis and visualization verification. OpenPose is a typical bottom-up 2D human pose estimation method, capable of detecting human keypoints and associating limbs through PartAffinityFields, and has been widely applied in multi-view human motion analysis tasks. Multi-view triangulation is a classic solution method in 3D reconstruction; its core idea is to recover the 3D coordinates of the same spatial point from multiple viewpoints based on the imaging constraints of the same point, given the known camera projection model.

[0082] S3.1, Multi-view 2D Human Keypoint Detection In the process of 3D human joint reconstruction, the accuracy of 2D keypoint detection directly affects the stability and accuracy of subsequent triangulation results. Therefore, this system first performs 2D human keypoint detection on the synchronized multi-view images. Let the i-th frame image under the i-th view be denoted as . After processing with OpenPose, the set of 2D human keypoints can be obtained as follows: (2-1) in, This represents the pixel coordinates of the j-th joint in the image plane. This represents the detection confidence level of the corresponding key point, where J is the total number of key points on the human body.

[0083] OpenPose uses convolutional neural networks to jointly predict keypoint heatmaps and limb association fields, thus achieving robust keypoint detection and human instance association in multi-person scenes. Considering that this system primarily collects data from single-person subjects, there is no need to further process complex cross-person association issues; it only needs to select the keypoint results corresponding to the main human target in each viewpoint. For abnormal detection results with low confidence or significant deviations from human structure, this system filters them before subsequent triangulation to reduce the impact of mismatches and noisy observations on 3D reconstruction.

[0084] To improve the usability of the 2D detection results in subsequent triangulation, this system performs the following preprocessing on key point observations: (1) Remove those with confidence scores below the threshold Key point observations.

[0085] (2) When there are obvious false positives or false negatives in individual perspectives, only the valid perspectives are retained to participate in subsequent triangulation.

[0086] (3) Combine the human body topology to manually inspect the abnormal points to ensure that the two-dimensional observation of the input triangulation process has good geometric consistency.

[0087] The effect obtained after the above treatment is as follows: Figure 9 As can be seen, the predicted location is basically correct.

[0088] In addition to qualitative and intuitive observations, this system also statistically analyzed the two-dimensional joint quality of different movements. The statistical results are shown in Table 5. It can be seen that the proportion of effective points for movements such as walking and turning, which completely obstruct the joints, is relatively low, while the average reliability of large-scale movements such as walking and exercise is relatively low.

[0089] Table 5. Quality Statistics of Two-Dimensional Key Point Detection Results

[0090] S3.2, Three-dimensional human joint restoration based on triangulation After obtaining two-dimensional keypoint observations from various perspectives, the three-dimensional human joint positions can be recovered by combining multi-camera calibration parameters. In multi-view geometry, the projection relationship from spatial points to image points can be expressed as: (2-2) in, Represents the homogeneous coordinates of a point in space. Let represent the homogeneous coordinates of the corresponding image point, s be the scale factor, and P be the camera projection matrix. For the v-th camera, its projection matrix can be written as: (2-3) Where K is the camera intrinsic parameter matrix, and These represent the rotation matrix and translation vector in the extrinsic parameters, respectively.

[0091] For the same joint j, if it is successfully detected in multiple viewpoints, then a three-dimensional view of the point can be constructed from the two-dimensional observations of each viewpoint. The system of linear constraint equations. Let the effective set of observation viewpoints be... Then the triangulation solution can be expressed as: (2-4) in, This represents the projection transformation from homogeneous coordinates to non-homogeneous pixel coordinates. The above solution essentially minimizes the reprojection error of a 3D point projected onto each viewpoint, which is a classic form of recovering the spatial point position in multi-view geometry.

[0092] Considering issues such as occlusion, detection errors, and mismatched viewpoints during actual data acquisition, this system introduces the following constraint strategies during the 3D human joint reconstruction process to improve the robustness of the triangulation results: (1) Only when the joint is not less than Triangulation recovery is only performed when the data is effectively detected from each perspective. (2) Two-dimensional observations participating in triangulation are screened according to the confidence level of key points, and high-confidence observations are retained first.

[0093] (3) Perform reprojection verification on the three-dimensional points obtained in the preliminary recovery. When the reprojection error of a certain view exceeds the threshold, the observation of that view is removed and the solution is recalculated.

[0094] (4) Manually check extreme abnormal frames to avoid local false detections from propagating to the subsequent SMPL surface truth construction process.

[0095] Through the above steps, the set of three-dimensional joints for each frame of the human body can be obtained: (2-5) S3.3, Validation and Quality Analysis of 3D Joint Results To verify the accuracy and usability of the three-dimensional human joint reconstruction results, this system analyzes them from two aspects: reprojection consistency and spatial structure rationality.

[0096] First, the reconstructed 3D joints are reprojected onto the image planes of each viewpoint and compared with the corresponding 2D keypoints. If the 3D reconstruction result is accurate, the reprojected joints should substantially coincide with the original 2D detection points in the image plane. Figure 10 As shown, Figure 10A visualization of the 3D human skeleton reconstruction results is shown, where (a) represents multi-view human image data acquired from various perspectives, and (b) represents the 3D human joint generation results obtained based on multi-view triangulation reconstruction. Figure 10 It can be seen that the recovered 3D skeleton fits the human body contour in the image well in terms of overall pose and joint positions, preliminarily verifying the effectiveness of the multi-view triangulation method. Based on this, the reprojection error of the j-th joint in frame t under the v-th view can be defined as: (2-6) Furthermore, the average reprojection error across all valid frames, all valid joints, and all valid viewpoints can be statistically analyzed as a quantitative indicator of the quality of 3D joint reconstruction. A smaller reprojection error indicates better consistency between the reconstructed 3D human joints and multi-view image observations.

[0097] Secondly, starting from the topological relationships of the human skeleton, a spatial rationality analysis is performed on the restored results. The three-dimensional joints of the human body should not only meet multi-view projection constraints but also conform to the real human movement patterns in their spatial distribution. For example, the positions of joints such as the left and right shoulders and hips should have a reasonable left-right symmetry, the lengths of upper and lower limb bone segments should remain relatively stable over a short period, and the overall skeletal structure should not exhibit obvious twisting, breakage, or crossing. For samples with abnormal topological structures, this system checks by retrospectively examining the two-dimensional detection results and camera projection relationships, and removes obviously abnormal frames from the subsequent SMPL optimization process to ensure the reliability of the training ground truth.

[0098] Furthermore, this system also provides intuitive verification of the 3D joint restoration effect through multi-view reprojection visualization results, such as... Figure 11 The experiment showed that the green skeleton represents the 2D joints generated by OpenPose, while the blue joints represent the reprojection results. The 3D skeletons of most samples fit the human body contour and joint positions well across all four viewpoints, indicating the practicality of the 3D human joint restoration process based on OpenPose and multi-view triangulation. Although individual joints may still shift under severe occlusion, local blurring, or extreme pose conditions, the overall results generally meet the pose prior requirements for subsequent parametric human model fitting.

[0099] In addition to the qualitative analysis and visualization results, this system also performed a quantitative analysis of the reprojection errors of the reconstructed 3D joint dataset, as shown in Table 6. Table 6: Statistics of Reprojection Errors in 3D Joint Reconstruction Results

[0100] Analysis of the reprojection errors reveals that walking movements have the largest errors, consistent with previous analysis. Furthermore, the two movements with the largest errors are precisely those with the lowest reliability of the average 2D joint point value, indicating that incorrect joint point predictions significantly impact triangulation and joint recovery. Overall, the average error for all movements is around 9 pixels. This is because the predicted 2D keypoints do not precisely fall on the actual pixel positions of the joints in every viewpoint. For multi-view triangulation, the key is whether there is reasonable geometric consistency between the observations from different views. Even if there are some pixel deviations in certain views, as long as the errors do not exhibit significant systematic shifts, the triangulation process can still utilize redundant information from multiple views to balance and compensate for the errors, thereby obtaining relatively accurate 3D joint estimates. Therefore, a large pixel error is not necessarily a bad thing; its actual impact still needs to be comprehensively judged in conjunction with the final 3D reconstruction results.

[0101] S4, A ground truth generation method for human body surfaces based on EasyMocap optimization While the obtained 3D human joints can effectively describe the overall posture and structure of the human body, they are essentially sparse representations, reflecting only the positional relationships of a small number of key skeletal points. This makes it difficult to directly meet the demand for dense supervised information in 3D human surface reconstruction tasks. For human surface reconstruction networks, the training phase requires not only skeletal information reflecting the overall posture but also mesh-level ground truth values ​​capable of characterizing human shape, limb contours, and local surface structures. Therefore, based on the 3D human joint reconstruction results, a parametric human model is further introduced to uniformly model human posture and shape, generating ground truth human surface values ​​that can be used for subsequent network training.

[0102] Considering the widespread application of the SMPL model in tasks such as human pose estimation, human reconstruction, and digital human modeling, this system selects it as the unified expression for the ground truth of human body surfaces. Meanwhile, EasyMocap provides a relatively complete multi-view human parameter optimization process, capable of solving human model parameters under multi-view image observation and 3D skeleton prior constraints. Therefore, this system adopts EasyMocap as the basic optimization framework for constructing the ground truth of human body surfaces.

[0103] S4.1, Parametric Human Model Representation and Surface Truth Value Requirements Analysis The ultimate goal of the human body 3D surface reconstruction task is to restore the continuous surface structure of the human body in 3D space. If only 3D joints are used as supervision signals, the model is prone to problems such as insufficient surface detail constraints and unstable mesh shape during training.

[0104] To address this issue, this system employs the SMPL as the ground truth representation of the human body surface. SMPL is a differentiable parametric human body model that generates human body surfaces for different individuals and poses using a fixed-topology human body template mesh and pose and shape-driven terms. A key advantage of the SMPL model is that it represents the human body in a unified parameter space, ensuring consistent vertex topological relationships across different samples. This facilitates its use in supervised learning tasks for vertex error calculation, joint error calculation, and mesh structure constraints.

[0105] Mathematically, let the set of vertices of the human body mesh be denoted as... Then the SMPL model can be represented as (2-7) in, Represents human body shape parameters. Represents human posture parameters. Indicates the global translation parameter. This represents the parameterized mapping function of SMPL.

[0106] After obtaining the vertices of the human body mesh, the corresponding model joint positions can be obtained further through the joint regression matrix, i.e.: (2-8) in, Let W represent the set of model joints, and W be a predefined joint regression matrix. Through this form, SMPL can achieve unified modeling between the human skeleton representation and the human surface representation, which facilitates subsequent supervised ground truth generation and network training.

[0107] Based on the needs of subsequent end-to-end human surface reconstruction training, this system ultimately aims to generate the following types of ground truth information for each frame sample: (1) True value of SMPL joint; (2) Truth values ​​of SMPL vertices; (3) True values ​​of SMPL parameters, including attitude parameters, shape parameters and global translation parameters.

[0108] S4.2, SMPL parameter optimization solution based on multi-view constraints After clarifying the representation of the true values ​​of the human body surface, the next step is to solve for the SMPL parameters based on multi-view observation data. This system uses EasyMocap to optimize and fit the parametric human body model. EasyMocap is designed for multi-view RGB video scenes and supports human body model reconstruction driven by 2D keypoints, 3D skeletons, and multi-view image information. Its repository description indicates that EasyMocap is designed for human motion capture tasks under various settings and can complete processes such as 3D skeleton reconstruction and SMPL fitting.

[0109] Based on the data preparation process of this system, the inputs for the SMPL parameter optimization process mainly include: (1) Synchronized multi-view image sequence (2) Multi-camera intrinsic and extrinsic parameters (3) Restored three-dimensional human joints During the optimization process, this system aims to find a set of optimal human model parameters so that the generated SMPL mesh fits the reconstructed human skeleton structure as closely as possible in 3D space, while its projection results remain consistent with human observations in multi-view images. To this end, an overall optimization objective can be constructed, incorporating 2D reprojection error, 3D joint error, pose prior constraints, and parameter regularization terms. Its general form can be expressed as: (2-9) in, This represents the error term between the model joints projected onto the multi-view image plane and the 2D keypoint detection results. This represents the error term between the model joint and the true value of the 3D joint. This represents the prior constraints on human posture. Regularization terms representing shape parameters and other model parameters. , , , The weights are assigned to each loss term.

[0110] The three-dimensional joint constraint term can be written as: (2-10) In the formula, This represents the position of the j-th model joint generated by the current SMPL parameters. This represents the ground truth value of the j-th 3D joint obtained by multi-view triangulation recovery from Section 3.1.

[0111] If we further consider the consistency of two-dimensional reprojection, the two-dimensional constraint terms can be written as: (2-11) in, Let v be the projection matrix of the v-th camera. This represents the two-dimensional observation position of the j-th joint from the v-th viewpoint. For the corresponding confidence weights, This represents the process of homogeneous projection onto pixel coordinates.

[0112] Under the combined effect of the above constraints, EasyMocap iteratively optimizes and solves for the human pose parameters, shape parameters, and global translation parameters for each frame. Finally, it outputs the SMPL human mesh vertices, model joint positions, and parameter results for the corresponding frame. Compared to fitting methods that rely solely on two-dimensional observations, this system introduces the three-dimensional joint priors recovered in Section 3.1, which can reduce the impact of multi-view detection errors and local occlusion on model parameter optimization to a certain extent, thereby improving the stability and reliability of the final ground truth of the human surface.

[0113] S4.3, SMPL Truth Coordinate Unification and Validation After optimizing the SMPL parameters, although the human body surface mesh and model joint results are obtained, these results still need to be further aligned with the coordinate system of subsequent training data before they can be directly used as network supervision signals. Therefore, this system further performs coordinate unification and result verification on the ground truth SMPL values.

[0114] In the data processing of this system, subsequent network training samples need to be in a unified reference coordinate system with the LiDAR human point cloud. Therefore, the SMPL joint and vertex results obtained through EasyMocap optimization need to be transformed from the reference camera coordinate system or world coordinate system to the LiDAR coordinate system based on the cross-sensor extrinsic calibration results. Let the mesh vertices in the reference coordinate system be... The corresponding grid vertices in the lidar coordinate system are Then its transformation relationship can be written as: (2-12) in, and These represent the rotation and translation transformations from the camera coordinate system to the LiDAR coordinate system, respectively. For model joint positions, the same extrinsic parameters can be used to achieve coordinate unification. After this processing, training and supervision fields corresponding one-to-one with the human point cloud input can be obtained.

[0115] To verify whether the constructed SMPL ground truth has sufficient accuracy and usability, this system analyzes the results from the perspective of image domain consistency.

[0116] First, in the image domain, the optimized SMPL joints or human body mesh is reprojected onto each camera image plane and compared with the original human body image observations. If the fitting results are accurate, the model joint projection positions should basically correspond to the key parts of the human body in the image, and the model contour should also maintain a high degree of consistency with the human body contour. In this way, it is possible to visually check whether there are obvious misalignments, incorrect limb orientations, or overall scale abnormalities in the human pose estimation.

[0117] observe Figure 12It was found that the model's joint projection position should correspond well with the key parts of the human body in the image, and the human body mesh outline should also be consistent with the real human body outline in terms of overall posture, limb direction and scale.

[0118] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for generating the true value of SMPL surfaces based on multi-view human joint constraints, characterized in that, Includes the following steps: Step 1: Build a multi-view human image acquisition system containing multiple cameras; Step 2: Use the calibration board to perform intrinsic parameter calibration, distortion parameter calibration and relative extrinsic parameter calibration on the multiple cameras respectively to obtain the projection matrix corresponding to each camera; Step 3: Acquire multi-view human body images at the same time, perform two-dimensional human body key point detection on each view image to obtain two-dimensional joint point observations containing pixel coordinates and confidence scores; determine the effective observation view set for each human body joint based on the confidence score and view validity screening rules of the two-dimensional joint point observations. Step 4: Based on the set of effective observation viewpoints and the projection matrix of each camera, perform multi-view triangulation on the same human joint to obtain the three-dimensional human joint. The three-dimensional points obtained from the initial recovery are reprojected for verification. When the reprojection error of a certain view exceeds the threshold, the observation of that view is discarded and the solution is recalculated. Step 5: Using the iteratively updated 3D human joints, multi-view 2D joint observations, and camera projection matrix as constraints, establish the SMPL parameter optimization model; By iteratively optimizing and solving the SMPL pose parameters, shape parameters, and global translation parameters, the SMPL human body surface vertices, SMPL model joints, and SMPL parameter ground values ​​are generated. Step 6: Perform reprojection consistency verification, human skeleton topology rationality verification, and coordinate unification verification on the generated SMPL surface ground values. Use the verification results as the training supervision ground values ​​for the human 3D surface reconstruction task.

2. The SMPL surface truth generation method based on multi-view human joint constraints as described in claim 1, characterized in that, Step 2 involves calibrating the intrinsic parameters, distortion parameters, and relative extrinsic parameters of the multiple cameras. Specifically, this includes: Step 21: Based on the checkerboard calibration board, solve the intrinsic parameter matrix of each camera by using the correspondence between the known positions of the corner points of the calibration board in physical space and their observed positions in the image. Step 22: After obtaining the corner point correspondence of each image, the initial value of the intrinsic parameter matrix is ​​first obtained by estimating the homography matrix. Then, based on this, nonlinear optimization is performed in combination with the distortion parameters to minimize the reprojection error of all corner points and solve for the radial distortion coefficient and tangential distortion coefficient. Step 23: Select one camera in the system as the reference camera. Based on the checkerboard calibration board, all cameras observe the same calibration board simultaneously. Establish cross-view correspondence using the same set of checkerboard corner points. Calculate the relative rotation matrix and translation vector from the reference camera to the m-th camera using the combination of extrinsic parameters of each camera relative to the world coordinate system. Under the condition of fixed intrinsic parameters, optimize the relative extrinsic parameters by minimizing the joint reprojection error between the reference camera and the m-th camera. Step 24: Organize the set of extrinsic parameters for the entire multi-camera system, with the reference camera as the center.

3. The SMPL surface truth generation method based on multi-view human joint constraints as described in claim 2, characterized in that, The specific process of solving the intrinsic parameter matrix of each camera in step 21 includes: Define the world coordinate system, camera coordinate system, image physical coordinate system, and pixel coordinate system; All corner points on the checkerboard calibration board are located on the same plane, and the third-dimensional coordinate of the corner points is always 0. A one-to-one correspondence is established between the known three-dimensional coordinates of the checkerboard corner points in physical space and their two-dimensional observation coordinates in the image. Based on the projection model, the projection relationship from the 3D coordinates of the checkerboard corner points to the pixel coordinates is simplified to the homography relationship from the plane to the image: Where s is the scale factor; Represents pixel coordinates; This represents the horizontal coordinate of the corner point of the chessboard grid in the calibration plate coordinate system (world coordinate system); This represents the vertical coordinate of the corner point of the chessboard grid in the calibration plate coordinate system (world coordinate system); H is the homography matrix from the calibration plate plane to the image plane; Different homography matrices correspond to different poses of the checkerboard image. Multiple homography matrices together impose constraints on the camera intrinsic parameters, thereby achieving intrinsic parameter estimation.

4. The SMPL surface truth generation method based on multi-view human joint constraints as described in claim 2, characterized in that, The objective function for minimizing the reprojection error of all corner points in step 22 is: in, This represents the actual detected corner pixel coordinates. This indicates the reprojection point estimated from the projection model based on the current parameters. and This represents the extrinsic parameter corresponding to the q-th calibration image; Q represents the number of calibration image groups, m represents the number of rows of the checkerboard corner points, K is the camera intrinsic parameter matrix, and D is the distortion parameter vector; i represents the row index of a corner point on the chessboard; j represents the column index of the corner point of the chessboard.

5. The SMPL surface truth generation method based on multi-view human joint constraints as described in claim 2, characterized in that, In step 23, when optimizing the relative extrinsic parameters, the objective function for the joint reprojection error between the reference camera and the m-th camera is: in, Let 'not' represent the actual corner pixel coordinates detected in the reference camera and the m-th camera. This indicates the corner coordinates obtained by projecting based on the current parameters; Indicates the distance from the reference camera coordinate system to The coordinate system of the m-th camera The relative rotation matrix; Indicates the distance from the reference camera coordinate system to The coordinate system of the m-th camera The relative translation vector; Indicates the index of the calibration image group.

6. The SMPL surface truth generation method based on multi-view human joint constraints as described in claim 1, characterized in that, Step 3 involves performing two-dimensional human keypoint detection on images from various viewpoints to obtain two-dimensional keypoint observations containing pixel coordinates and confidence levels. Specifically, this includes: Two-dimensional human keypoint detection is performed on the synchronized multi-view images. A convolutional neural network is then used to jointly predict the keypoint heatmap and limb association field, outputting a set of two-dimensional human keypoints. in, This represents the pixel coordinates of the j-th joint in the image plane. This represents the detection confidence level of the corresponding key point, where J is the total number of human key points; Let v represent the set of two-dimensional key points of the human body detected from the v-th viewpoint at time t; Preprocessing of 2D key point observations: remove key point observations with confidence levels below the preset threshold τ, and screen and filter abnormal detection results in combination with human body topology. When false detections or missed detections occur in individual perspectives, only valid perspectives are retained for subsequent triangulation.

7. The SMPL surface truth generation method based on multi-view human joint constraints as described in claim 1, characterized in that, The objective function of the SMPL parameter optimization model established in step 5 is: in, This represents the error term between the model joints projected onto the multi-view image plane and the 2D keypoint detection results. This represents the error term between the model joint and the true value of the 3D joint. This represents the prior constraints on human posture. Regularization terms representing shape parameters and other model parameters. , , , Assign weights to each loss term; The three-dimensional joint constraints are: In the formula, This represents the position of the j-th model joint generated by the current SMPL parameters. This represents the ground truth value of the j-th 3D joint obtained by multi-view triangulation. The two-dimensional reprojection constraint terms are: in, Let v be the projection matrix of the v-th camera. This represents the two-dimensional observation position of the j-th joint from the v-th viewpoint. For the corresponding confidence weights, This represents the process of homogeneous projection onto pixel coordinates.

8. The SMPL surface truth generation method based on multi-view human joint constraints as described in claim 1, characterized in that, Step 6 involves verifying the coordinate uniformity of the generated SMPL surface true values, specifically including: Based on the cross-sensor extrinsic calibration results, the SMPL joints and vertices are transformed from the camera coordinate system to the lidar coordinate system: in, and These represent rotation and translation transformations from the camera coordinate system to the lidar coordinate system, respectively. Where J_C and V_C represent the joint and vertex coordinates of the SMPL model in the camera coordinate system, respectively; J_L and V_L represent the joint and vertex coordinates of the SMPL model in the lidar coordinate system, respectively; and R_CL and t_CL represent the rotation matrix and translation vector from the camera coordinate system to the lidar coordinate system, respectively.

9. A SMPL surface truth generation system based on multi-view human joint constraints, characterized in that, include: The multi-view image acquisition module is used to build a multi-view human body image acquisition system containing multiple cameras to acquire multi-view human body images at the same time. The camera calibration module is used to perform intrinsic parameter calibration, distortion parameter calibration and relative extrinsic parameter calibration on the multiple cameras using a calibration board, so as to obtain the projection matrix corresponding to each camera; The two-dimensional joint detection and effective viewpoint selection module is used to perform two-dimensional human keypoint detection on images from various viewpoints, obtain two-dimensional joint observations containing pixel coordinates and confidence scores, and determine the effective observation viewpoint set for each human joint based on the confidence scores and viewpoint effectiveness selection rules of the two-dimensional joint observations. The three-dimensional joint iterative recovery module is used to perform multi-view triangulation on the same human joint based on the effective observation view set and the projection matrix of each camera to obtain the three-dimensional human joint, and to perform reprojection verification on the initially recovered three-dimensional points. When the reprojection error of a certain view exceeds the threshold, the observation of that view is discarded and the solution is re-solved. The SMPL parameter joint optimization module is used to establish an SMPL parameter optimization model with the iteratively updated 3D human joints, multi-view 2D joint point observations and camera projection matrix as constraints. Iterative optimization solves the SMPL pose parameters, shape parameters and global translation parameters, and generates SMPL human surface vertices, SMPL model joints and SMPL parameter ground values. The triple verification and supervised ground truth generation module is used to perform reprojection consistency verification, human skeleton topological rationality verification, and coordinate unification verification on the generated SMPL surface ground truth. The verification results are used as the training supervised ground truth for the human 3D surface reconstruction task.

10. The SMPL surface truth generation system based on multi-view human joint constraints as described in claim 9, characterized in that, The camera calibration module includes: The intrinsic parameter calibration submodule is used to solve the intrinsic parameter matrix and distortion parameters of each camera based on the chessboard calibration board and the correspondence between the known positions of the corner points of the calibration board in physical space and their observed positions in the image. The extrinsic parameter calibration submodule is used to select one camera in the system as a reference camera. Based on the checkerboard calibration board, it uses the same set of checkerboard corner points when each camera observes the same calibration board at the same time to calculate the relative rotation matrix and translation vector from the reference camera to the m-th camera. The calibration and verification submodule is used to calculate the average reprojection error on all calibration images of a single camera, the average joint reprojection error between the reference camera and the m-th camera, and to draw the actual detected corner points and reprojected corner points in the calibration images for visual comparison. Based on the external parameter calibration results, the relative positions of each camera are restored and the spatial structure consistency is verified with the actual deployment scene.

Images