A method and system for constructing a visual-aided lidar human surface reconstruction dataset

Abstract

This invention discloses a method and system for constructing a visually assisted LiDAR human surface reconstruction dataset. The method constructs a collaborative acquisition system of LiDAR and multi-view cameras, establishes a unified reference system through spatial calibration, and achieves temporal matching of point cloud frames and image frames based on cross-host time offset measurement, fixed compensation, and threshold filtering. It recovers the three-dimensional joints of the human body using multi-view images and fits an SMPL model to generate surface ground truth, which is then transformed to the LiDAR coordinate system. The original point cloud is cropped, ground plane removed, clustered, and jointly cleaned to form multiple versions of point cloud input. Invalid samples are removed through spatial consistency checks, and the data is encapsulated into standardized training data containing point clouds, pose parameters, shape parameters, three-dimensional joints, surface vertices, and auxiliary labels. This invention improves the consistency and reliability of cross-modal training samples and is suitable for data construction of LiDAR human surface reconstruction models.

Description

Technical Field

[0001] This invention relates to the fields of three-dimensional perception, computer vision, lidar point cloud processing, and deep learning training data construction technology, and in particular to a method and system for constructing a visual-assisted lidar human body surface reconstruction dataset. Background Technology

[0002] 3D reconstruction of the human body surface aims to restore the posture, shape, and surface mesh structure of the human body through sensor observation, and has significant application value in fields such as rehabilitation assessment, motion analysis, intelligent interaction, digital twins, and virtual reality. Existing methods largely rely on monocular or multi-view visual images for human body mesh reconstruction. These methods can utilize rich appearance texture and human contour information, but they are also easily affected by changes in lighting, occlusion, background complexity, and privacy protection requirements. At the same time, simple image observation is essentially a two-dimensional projection, and the depth scale and spatial structure of the human body surface need to be indirectly restored by relying on model priors or learning results, which may lead to spatial deviations in complex scenes.

[0003] As an active 3D measurement device, LiDAR can directly obtain spatial point clouds of target surfaces. Compared with pure vision sensors, it has advantages such as not relying on ambient lighting, clear spatial geometric information, and lower privacy exposure. Therefore, using LiDAR point clouds for human surface reconstruction has good engineering application potential. However, the effective sampling points of the human body in LiDAR point clouds are usually sparse, and point clouds suffer from problems such as disorder, uneven density, local occlusion, ground background interference, and outlier noise. This makes it difficult to obtain stable supervised constraints when directly training human surface reconstruction networks using raw point clouds.

[0004] For end-to-end human surface reconstruction models, high-quality training samples require not only LiDAR human point cloud input, but also the corresponding 3D joints, human pose parameters, shape parameters, and surface mesh ground truth. Traditional ground truth construction methods often rely on optical motion capture systems, inertial sensors, or manual annotation. Optical motion capture systems have high requirements for site and equipment conditions, inertial sensors require subjects to wear additional devices, and manual annotation is difficult to obtain dense surface-level ground truth, all of which are not conducive to the construction of low-cost, scalable datasets.

[0005] Multi-view vision systems can acquire human images from different directions and generate relatively complete 3D human supervision information through 2D joint detection, multi-view geometric constraints, and parametric human model fitting. If the multi-view vision system can provide supervision ground truth during the training data generation stage, and then the generated ground truth can be registered and encapsulated with the LiDAR human point cloud at the same time, it is possible to balance the annotation capabilities of the vision system with the privacy and illumination robustness advantages of the LiDAR deployment stage.

[0006] However, in actual data acquisition, LiDAR and multiple cameras typically have different coordinate systems, sampling frequencies, data formats, and host time bases. Existing methods do not provide reliable spatial calibration and temporal alignment schemes. Without such calibration, the point cloud input and the visual supervision ground truth will not correspond to the same instant of human pose, leading to training label mismatch. Furthermore, the original LiDAR point cloud contains ground, environmental structures, outliers, and local clustering failures. Existing methods lack point cloud purification and invalid sample removal mechanisms for training sample quality control. Therefore, there is an urgent need for a data generation method that can uniformly complete cross-sensor spatial calibration, cross-device temporal compensation, human point cloud extraction and purification, supervision ground truth mapping, consistency checks, and training sample encapsulation. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a method and system for constructing a visually assisted LiDAR human surface reconstruction dataset. This method solves the problems of difficulty in obtaining high-quality 3D supervised ground truth, difficulty in accurately corresponding cross-modal observations, strong background interference of original point clouds, and non-standard organization of training samples in the existing LiDAR human surface reconstruction training process.

[0008] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a method for constructing a visually assisted lidar human surface reconstruction dataset, comprising the following steps: Step 1: Construct a laser-visual collaborative acquisition system, which includes a laser radar and a multi-view camera; acquire 3D point clouds of the human body and environment through the laser radar, and acquire human body images from different perspectives through the multi-view camera; record and store the point clouds, the images and their respective timestamps in a unified manner. Step 2: Spatial calibration is performed on the lidar and the multi-view camera, and a cross-sensor spatial relationship is established using a calibration auxiliary device; the pose of the calibration auxiliary device in the camera coordinate system is solved using the multi-view camera, and the pose of the calibration auxiliary device in the lidar coordinate system is solved using the lidar; based on the pose correspondence between the calibration auxiliary device in the camera coordinate system and the lidar coordinate system at the same moment, the extrinsic parameters of the lidar to the camera are solved; Step 3: Time synchronization and frame matching are performed between the lidar and the multi-view camera. The image timestamps of the multi-view camera are compensated and corrected with the time reference of the lidar as a reference. The point cloud frames of the lidar are used as a reference to match the image frames that are closest in time to each point cloud in the compensated image sequence, and effective point cloud-image sample pairs are selected by time threshold. Step 4: Construct multi-source supervision information and human point cloud samples. Organize the LiDAR point cloud, multi-view camera image, and human surface supervision information at the same time after spatiotemporal alignment into the same sample level. Perform human region cropping, ground removal based on random sampling consensus algorithm, density-based spatial clustering purification, and combined purification of ground removal and clustering purification on the original point cloud to form four human point cloud input versions with different purification levels: cropped version, ground removal version, clustering version, and combined purification version. Transform the human surface supervision information into the LiDAR coordinate system and perform spatial consistency checks with each version of the human point cloud input to remove invalid samples. Step 5: Package the training samples and organize the dataset. Package the valid samples into standardized training samples. The standardized training samples include human point clouds, pose parameters, shape parameters, 3D joints, surface vertices and auxiliary labels. Divide the dataset into training set, validation set and test set according to the acquisition sequence to form a dataset for training the human surface reconstruction model.

[0009] Further, in step 2, the calibration auxiliary device is a checkerboard calibration board; the step of solving the pose of the calibration auxiliary device in the camera coordinate system using the multi-view camera includes: using camera intrinsic parameters, detecting the image corner points of the checkerboard calibration board, and combining the known checkerboard spatial coordinates to solve the rotation matrix and translation vector of the checkerboard calibration board relative to the camera coordinate system; the step of solving the pose of the calibration auxiliary device in the lidar coordinate system using the lidar includes: extracting the point cloud corresponding to the surface of the checkerboard calibration board from the original point cloud, performing plane fitting on the extracted point cloud to obtain the plane normal vector and plane offset, and recovering the pose of the checkerboard calibration board in the lidar coordinate system based on the geometric center of the board surface point cloud and the plane normal vector.

[0010] Furthermore, in step 2, observation data of the checkerboard calibration board under multiple different postures are collected, and a joint optimization model is constructed to solve the extrinsic parameters of the lidar to the camera. The goal of the joint optimization model is to ensure that the pose transformation relationship of the checkerboard calibration board in the camera coordinate system and the lidar coordinate system remains consistent in each set of observation data.

[0011] Further, in step 3, the compensation and correction of the image timestamps of the multi-view camera specifically includes: using the host connected to the lidar as the time reference end and the host connected to the multi-view camera as the end to be compensated; measuring the time difference between the two hosts multiple times before acquisition, statistically analyzing the multiple measurement results, and selecting the median or truncated mean as a fixed time offset; using the fixed time offset to uniformly compensate the image timestamps of the multi-view camera; setting a time threshold, comparing the time difference between the point cloud frame and its matched image frame with the time threshold, retaining the point cloud-image pair as a valid sample pair when the time difference is less than or equal to the time threshold, and discarding the pair when the time difference is greater than the time threshold; the time threshold is selected based on the statistical results of the sample retention rate under different threshold values.

[0012] Furthermore, the time threshold is determined comprehensively based on the clock synchronization accuracy of the acquisition system, the speed of human movement, and the sample retention rate. When the acquisition frame rate of the lidar is 10Hz and the acquisition frame rate of the multi-view camera is 30Hz, the value range of the time threshold is 15 milliseconds to 50 milliseconds.

[0013] Further, in step 4, the human body surface supervision information is generated in the following way: using the multi-view camera image after time synchronization, the three-dimensional joints of the human body are recovered through two-dimensional joint detection and multi-view geometric constraints; the three-dimensional joints of the human body are used to fit a parameterized human body model to generate posture parameters, shape parameters and surface vertices; the generated posture parameters, shape parameters and surface vertices are transformed from the camera coordinate system to the lidar coordinate system through the extrinsic parameters obtained in step 2; the parameterized human body model is an SMPL model.

[0014] Furthermore, in step 4, the spatial consistency check includes: superimposing the human body point cloud input of each version with the surface vertices transformed into the lidar coordinate system in the same coordinate system to check the overall position consistency and posture direction consistency; and marking samples with incomplete human body after clustering, or with local regional offsets or inconsistent limb directions between the human body point cloud input and the surface vertices as invalid samples and removing them.

[0015] Furthermore, in step 5, the step of dividing the training set, validation set, and test set according to the overall acquisition sequence includes: dividing the data by acquisition sequence to ensure that all samples of the same acquisition sequence belong to the same data subset; stratifying the data according to different action categories and weighting the allocation within each action category according to the number of sequence frames; and dividing all valid sequences into the training set, validation set, and test set according to a preset ratio of eight to one to one.

[0016] The present invention provides a system for constructing a visually assisted lidar human surface reconstruction dataset, used to execute the above method, the system comprising: The laser-visual collaborative acquisition unit includes a laser radar, a multi-view camera, a data acquisition and storage terminal, and a calibration auxiliary device. It is used to acquire three-dimensional point clouds of the human body and the environment, human body images from different perspectives, and to uniformly record and store the point clouds, the images, and their timestamps. A spatial calibration unit is used to establish cross-sensor spatial relationships using the calibration auxiliary device and to solve for the extrinsic parameters of the laser radar to the camera. The time synchronization and frame matching unit is used to compensate and correct the image timestamps of the multi-view camera with the time reference of the lidar, and to filter valid point cloud-image sample pairs with the point cloud frames of the lidar as a reference through a time threshold. The data generation and sample construction unit is used to organize the spatiotemporally aligned LiDAR point cloud, the multi-view camera image, and the human surface supervision information into the same sample level. It performs human region cropping, ground removal based on random sampling consensus algorithm, density-based spatial clustering purification, and combined purification of ground removal and clustering purification on the original point cloud to form four human point cloud input versions with different purification levels: cropped version, ground removal version, clustered version, and combined purification version. It also converts the human surface supervision information to the LiDAR coordinate system and performs spatial consistency checks to remove invalid samples. The dataset organization unit is used to encapsulate valid samples into standardized training samples and divide them into training, validation, and test sets according to the collection sequence to form a dataset for training human surface reconstruction models.

[0017] Furthermore, in the data generation and sample construction unit, the human body surface supervision information is generated in the following way: using the multi-view camera image after time synchronization, the three-dimensional joints of the human body are recovered through two-dimensional joint detection and multi-view geometric constraints; the three-dimensional joints of the human body are used to fit the SMPL model to generate posture parameters, shape parameters and surface vertices; the generated posture parameters, shape parameters and surface vertices are transformed from the camera coordinate system to the lidar coordinate system through the extrinsic parameters obtained by the spatial calibration unit.

[0018] The beneficial effects of this invention are as follows: This invention provides a method for constructing a visually assisted LiDAR human surface reconstruction dataset. The method constructs a collaborative acquisition system of LiDAR and multi-view cameras, establishing a unified spatial reference relationship through camera calibration and LiDAR-camera extrinsic parameter calibration. It achieves temporal matching of point cloud frames and image frames based on cross-host time offset measurement, fixed compensation, and time threshold filtering. The method utilizes multi-view images to recover the three-dimensional joints of the human body and generate parameterized ground truth values ​​for the human surface, which are then transformed to the LiDAR coordinate system. The original point cloud undergoes cropping, ground removal, clustering purification, and joint purification processes to form multiple versions of the point cloud input. Invalid samples are eliminated through spatial consistency checks between the point cloud and the ground truth values, and the valid samples are encapsulated into standardized training data containing point clouds, pose parameters, shape parameters, three-dimensional joints, surface vertices, and auxiliary labels. This invention improves the consistency, reliability, and traceability of cross-modal training samples and is suitable for data construction of LiDAR human surface reconstruction models. Compared with existing technologies, this invention has at least the following beneficial effects: First, by constructing a laser-visual collaborative acquisition system that includes LiDAR and multi-view cameras, high-quality human surface supervision ground truth values ​​are generated using the multi-view vision system during the training data generation stage. At the same time, LiDAR point clouds are used as the main input of the model, which reduces the problem of lack of dense labels in pure LiDAR data and achieves a balance between the labeling capabilities of the vision system and the privacy and illumination robustness advantages of LiDAR deployment.

[0019] Second, by using LiDAR to calibrate the space of multi-view cameras, a calibration board is used to establish cross-sensor spatial relationships, unifying LiDAR point clouds and visual supervision results under the same spatial reference system. At the same time, through cross-host time offset measurement, fixed compensation, and time threshold filtering strategies, point cloud frames and image frames are mapped to the same instant of human movement, overcoming the spatiotemporal mismatch problem caused by different coordinate systems, sampling frequencies, and acquisition host time references of heterogeneous sensors, and significantly improving the spatiotemporal consistency of cross-modal training samples.

[0020] Third, by performing human body region cropping, ground removal based on the random sampling consensus algorithm, density-based spatial clustering purification, and combined purification of ground removal and clustering purification on the original point cloud, four human body point cloud input versions with different purification levels are formed: cropped version, ground removal version, clustered version, and combined purification version. This effectively reduces the interference of ground points, background structures, outliers, and local clustering failures on model training, and provides a comparable data basis for analyzing the impact of different preprocessing strategies on training results.

[0021] Fourth, by converting the human body surface supervision information generated by multi-view vision to the lidar coordinate system and performing spatial consistency checks with the human body point cloud input of each version, samples with incomplete human body subjects after clustering or with pose misalignment or position offset between point cloud and surface ground value are marked as invalid and removed, effectively ensuring the geometric correctness and reliability of training samples.

[0022] Fifth, by encapsulating the effective samples into standardized training samples containing human point clouds, pose parameters, shape parameters, 3D joints, surface vertices, and auxiliary labels, and dividing the training set, validation set, and test set according to the overall acquisition sequence, the data leakage problem caused by the cross-set distribution of adjacent frames is avoided, and the callability, traceability, and reliability of model evaluation of the dataset are improved.

[0023] Sixth, the data obtained by this invention can support a model training paradigm of visual-assisted training and independent inference by LiDAR. During the training phase, dense supervised ground truth is generated using multi-view vision, and during the deployment or inference phase, it only relies on LiDAR point cloud input. This is beneficial for deploying human surface reconstruction models in privacy-sensitive or lighting-complex scenarios, thus broadening the application scope of LiDAR human surface reconstruction technology.

[0024] The above and other objects, advantages, and features of the present invention will be more fully set forth and demonstrated through the following detailed description of specific embodiments in conjunction with the accompanying drawings. Those skilled in the art, upon referring to the following detailed description and the accompanying drawings, will be able to better understand and realize the above advantages of the present invention. Other objects, features, and advantages of the present invention will become clearer after being described in detail in the detailed description section in conjunction with the accompanying drawings. Attached Figure Description

[0025] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following drawings are provided for illustration.

[0026] Figure 1 This is a schematic diagram of the data acquisition system scenario; Figure 2 The workflow of the Leishi Collaborative Data Acquisition System; Figure 3 For cross-sensor reprojection results; Figure 4 For cross-sensor reprojection results; Figure 5 A flowchart for time synchronization and matching process; Figure 6 A table showing the statistical results of cross-host time difference for different groups of data; Figure 7 A schematic diagram showing the sample retention rate under different time thresholds; Figure 8The statistical results of the time difference for valid samples with a 15ms threshold; Figure 9 The statistical results of the time difference for valid samples with a 20ms threshold; Figure 10 The statistical results of the time difference for valid samples with a 30ms threshold; Figure 11 The statistical results of the time difference for valid samples with a 50ms threshold; Figure 12 To synchronize data from four rear cameras for display; Figure 13 Display of LiDAR data after time synchronization; Figure 14 This is a diagram illustrating DBSCAN clustering. Figure 15 This is a schematic diagram of the processing flow for four datasets; Figure 16 A comparison chart of data processing; Figure 17 Comparison of detailed data sets plane and clean; Figure 18 To check the mapping relationship between preprocessed point clouds and cameras; Figure 19 To check the mapping relationship between the preprocessed point cloud and the SMP truth value; Figure 20 Examples of invalid samples with poor clustering results; Figure 21 This is an example of an invalid sample with inconsistent pose. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0028] Example 1

[0029] This embodiment provides a method for constructing a visually assisted LiDAR human surface reconstruction dataset, which specifically includes the following steps: S1, Design of a synchronous acquisition system for LiDAR and RGB camera. A system was constructed as follows... Figure 1 As shown, Figure 1 This is a schematic diagram of the data acquisition system, which consists of a lidar system, a multi-view camera, a data acquisition and storage terminal, and a calibration auxiliary device. The system uses lidar as the primary device for acquiring three-dimensional geometric information of the human body, and a multi-view camera as an auxiliary visual perception device. Through a unified data recording, organization, and representation method, it provides the raw data foundation for subsequent training sample construction and model validation.

[0030] From a system composition perspective, the LiDAR module is primarily responsible for acquiring 3D point cloud observations of the human body in space, serving as the main input source for subsequent 3D surface reconstruction models. The multi-view camera module is used to simultaneously acquire human image information from different perspectives, providing auxiliary support for subsequent multi-view geometric constraint establishment, training label generation, and cross-modal data association. The data acquisition and storage terminal is responsible for the unified storage and management of point cloud data, image data, and their associated information. The calibration auxiliary device is used to support the establishment of spatial relationships between multiple sensors. These modules together constitute a LiDAR-Vision collaborative perception platform for 3D surface reconstruction tasks. The platform's specific workflow is as follows: Figure 2 As shown, Figure 2 The following describes the workflow of the Ravis Collaborative Data Acquisition System. The main sensor parameters of the system are shown in the table.

[0031] Table 1 Main parameters of the sensors in the data acquisition system

[0032] Table 2. Deviation between LiDAR projection and camera detection corner points

[0033] Table 2 presents the average reprojection error and standard deviation of the four cameras under multiple sets of calibration data. The statistical results show that the overall projection error of the LiDAR to the camera extrinsic parameter calibration is between 3 and 5 pixels, which is considered a good result. This indicates that the obtained extrinsic parameters can effectively establish the spatial correspondence between the point cloud coordinate system and the image coordinate system. Figure 3 and Figure 4 As shown, after projecting the LiDAR point cloud onto the corresponding viewpoint image based on the calibration results, the point cloud contour basically matches the human body area and the corner points of the checkerboard calibration board in the image, further verifying the accuracy of the extrinsic parameter calibration and meeting the needs of subsequent point cloud projection, cross-modal data association, and training sample construction.

[0034] S3, a multi-device time synchronization method, addresses the challenges of establishing accurate correspondences between multi-source heterogeneous data in laser-visual collaborative data acquisition systems. In such systems, LiDAR and multi-view cameras are typically driven independently by different computing devices, with each device generating timestamps from its local system clock. However, differences in clock drift between different hosts during startup, system timing, and operation make it difficult for data from different sensors to naturally reside on the same time base. Network Time Protocol (NTP), a widely adopted clock synchronization mechanism in distributed computing environments, enables clock alignment across multiple host systems in a network environment and achieves high synchronization accuracy in low-latency environments such as local area networks. For human 3D surface reconstruction, time consistency is fundamental for establishing accurate correspondences between multi-source heterogeneous data. When a human body is in motion, its posture, limb positions, and surface morphology change continuously over time. If there is a discrepancy between the LiDAR point cloud and the multi-view image acquisition time, even with accurate spatial extrinsic calibration, cross-modal projection may still exhibit phenomena such as contour misalignment, local edge shifts, or non-overlapping of key human body parts. The performance of LiDAR and camera fusion methods is highly dependent on the quality of spatiotemporal alignment. Temporal and spatial mismatch between sensors will directly weaken cross-modal registration and fusion effects.

[0035] The importance of time synchronization for the subsequent dataset construction process of this system is mainly reflected in two aspects: First, in the verification process of projecting LiDAR point clouds onto the image plane, the projection result can only truly reflect the quality of extrinsic parameter calibration when the point cloud frame and the image frame correspond to approximately the same sampling time; otherwise, the time deviation will couple with the spatial calibration error, interfering with the extrinsic parameter verification conclusion. Second, after recovering the 3D joint points using multi-view images and further generating SMPL ground truth values, it is necessary to pair them with the LiDAR human body point cloud at the corresponding time to form supervised samples. If the time axis is not aligned, the training input and the supervised label will no longer satisfy the observation relationship at the same time.

[0036] From a methodological perspective, cross-device time synchronization needs to meet the following requirements: First, a unified and stable time reference should be established among multiple acquisition hosts to ensure the comparability of timestamps generated by different devices; second, synchronization errors should be controlled within an acceptable range for human dynamic acquisition tasks, thereby supporting reliable matching of image frames and point cloud frames; third, system deployment complexity should be minimized to avoid modifications to the existing acquisition process due to additional hardware triggering circuits. Software solutions based on system clock synchronization can support collaborative acquisition by multiple device sensors, improving the consistency of multi-sensor data without significantly increasing system complexity. Considering factors such as the experimental platform consisting of multiple ordinary computing devices, the acquisition environment being a local area network, and system implementation costs, an NTP-based software time synchronization solution was chosen as the method for establishing a unified time reference across devices. Figure 5As shown, the time synchronization and frame matching process of this system mainly includes three stages: cross-host time difference measurement and fixed offset compensation, nearest neighbor matching of point cloud frames and image frames, and effective sample pair screening based on time threshold. These are described in detail below.

[0037] The Rayvision collaborative data acquisition system consists of a Linux host connected to the LiDAR and a Windows host connected to four cameras. Point cloud and image data are independently acquired and timestamped on different devices. Therefore, the consistency of the cross-device time reference directly affects the accuracy of subsequent sample pairing. If relying entirely on real-time time synchronization results in the network environment, network transmission latency and its fluctuations may introduce additional uncertainties, thus affecting the stability of time synchronization. A software synchronization strategy of "first measuring the fixed time difference between devices, and then compensating the client timestamps" is adopted to reduce the impact of instantaneous network fluctuations on cross-device data association. NTP provides a basic basis for establishing a unified time reference between different hosts. Its clock offset estimation idea comes from the standard four-timestamp interaction model, providing a methodological basis for subsequent time difference measurement and compensation.

[0038] In the implementation, a Linux host connected to the LiDAR is used as the time reference, and a Windows host connected to four cameras is used as the compensation target. Before data acquisition begins, multiple time difference queries are performed on both hosts in a local area network environment to obtain a set of time deviation measurement results between the Windows host and the Linux host. Since a single query result is affected by factors such as network propagation latency and system scheduling jitter, there are usually fluctuations between different measurements; therefore, a single measurement result cannot be directly used as the final compensation amount. To improve the stability of the compensation, this system performs statistical analysis on multiple measurement results and selects a fixed offset that can represent the true time deviation between the two hosts. This is used to uniformly compensate for the client's collected timestamps in subsequent steps. The core idea behind this is to transform the time synchronization problem in the collection phase into a relatively stable "fixed time difference compensation" problem, thereby avoiding the direct transmission of instantaneous network disturbances to the sample pairing results.

[0039] Let the Linux host timestamp be The original timestamp of the Windows host is After completing the time difference measurement, the compensation form for the client image timestamp can be expressed as: (1-26) in, This represents the compensated image timestamp. This represents a fixed time offset selected based on multiple time difference measurements from two main units. When When, it indicates that the client's original timestamp needs to be forward-compensated; when If the original timestamp of the client needs to be compensated backwards, it means that the client's original timestamp needs to be compensated backwards. The essence of this process is to map the image time recorded on the Windows host to a reference timeline consistent with that of the Linux host, so that the two are comparable in subsequent processing.

[0040] In order to obtain stability The estimated value is the result of multiple time difference queries in this system. Statistical analysis is performed, and a compensation value that better reflects the overall deviation level is selected from candidate results such as the mean, median, or truncated mean. Let the k-th measurement result be denoted as... Then its statistical mean can be expressed as: (1-27) In the presence of a few abnormally fluctuating measurements, the median or truncated mean is generally more stable than a single measurement, and therefore more suitable as a candidate estimate for the fixed compensation amount. Ultimately, this system combines the central tendency and fluctuation of multiple measurement results to select a suitable median or mean. This serves as a uniform time compensation measure used throughout the entire acquisition phase. Thus, during a single acquisition task, there's no need to repeatedly and dynamically modify the timestamp based on real-time network conditions; instead, a fixed offset compensation is applied to the client-side image timestamp.

[0041] After completing timestamp compensation, this system further establishes the time matching relationship between image frames and point cloud frames. Since the four cameras acquire images at 30Hz, while the LiDAR acquires point clouds at 10Hz, there is a significant difference in sampling frequency between the two types of data. Therefore, this system adopts a "high-density to low-density alignment" matching strategy, that is, using the low-frequency LiDAR point cloud frame as a benchmark, it searches for the image frame in the high-frequency image sequence that is temporally closest to each point cloud frame. Let the timestamp of the j-th LiDAR point cloud frame be... The timestamp of the compensated image in the i-th frame is The time difference between the two is defined as: (1-28) For a given point cloud frame, its corresponding image frame is determined by the minimum time difference criterion, i.e.: (1-29) This allows us to obtain a one-to-one matching candidate relationship between point cloud frames and image frames. By adopting this method of aligning from low-density sequences to high-density sequences, we can ensure that each frame of LiDAR point cloud can find a nearest temporal reference in the image sequence, which better meets the needs of constructing training samples using point clouds as the main input.

[0042] However, matching based solely on the minimum time difference is insufficient to guarantee that sample pairs are sufficiently close in time. Therefore, this system further introduces a time threshold. The matching results are filtered. The minimum time difference between a LiDAR point cloud frame and its nearest image frame satisfies... (1-30) If the time threshold is too high, the point cloud frame and the image frame are considered to constitute a valid cross-modal sample pair; otherwise, the match is discarded as the time deviation is too large. Considering that a large time threshold, while retaining more samples, reduces the accuracy of the correspondence between the image and point cloud pairs at the same instant of human motion, and a small time threshold, while providing stronger temporal consistency, leads to a significant decrease in the number of valid samples, this system uses sample retention rate statistics under different time thresholds to help determine the final matching threshold. If the threshold is within the acceptable range... The number of valid matching samples successfully retained is If the total number of original point cloud frames is N, then the sample retention rate corresponding to this threshold can be expressed as: (1-31) By comparing the retention rate changes corresponding to different time thresholds, and considering the time consistency requirements and sample size requirements, a suitable threshold that can balance matching accuracy and sample size is selected for the final time matching.

[0043] To ensure the traceability of subsequent data processing, this system simultaneously retains the original image timestamp, the compensated image timestamp, the LiDAR timestamp, the matching index between the point cloud and the image using the fixed offset, the minimum time difference, and the matching statistics under different thresholds during the sample construction stage, and writes them all into the matching result file. This provides a basis for outlier removal, adjustment of time compensation strategies, and threshold sensitivity analysis, and also facilitates tracking the impact of time alignment on the final sample quality during point cloud projection verification, SMPL ground truth association, and training sample organization.

[0044] To verify the effectiveness of the cross-device time compensation and sample matching method adopted in this system, it is necessary to evaluate it from two aspects: the stability of time compensation and the quality of cross-modal sample pairing. The former focuses on whether the fixed time offset between hosts obtained through multiple time difference queries before acquisition has sufficient stability and can be used as a uniform compensation amount throughout the entire acquisition phase. The latter focuses on whether, after completing fixed offset compensation, a reliable matching relationship can be established between the LiDAR point cloud frames and image frames under a unified time axis, and whether a sufficient number of valid samples are retained while ensuring time consistency. Therefore, this system comprehensively evaluates the time synchronization effect from three aspects: fixed time offset statistics, sample retention rate analysis under different time thresholds, and visualization verification of the final matched samples.

[0045] In terms of fixed time offset evaluation, this system performs multiple time difference queries on the Linux host connected to the LiDAR and the Windows host connected to four cameras before formal data acquisition, obtaining a set of cross-host time offset measurement results. Since a single measurement is affected by factors such as network propagation latency, system scheduling delay, and fluctuations in time service response, there is usually some dispersion between different measurement results. Therefore, this system statistically analyzes the minimum, maximum, mean, median, and standard deviation of the time difference sample sequence to determine whether the time offset between the two hosts exhibits a relatively stable, concentrated distribution. If the multiple measurement results fluctuate within a small range, and the mean and median are close, it indicates that there is a stable time difference between the two hosts that can be approximated by a fixed offset. Conversely, if the measurement results are too dispersed, it means that a simple fixed compensation strategy is insufficient to accurately characterize the cross-device time relationship. Based on this statistical result, this system selects a more representative fixed offset from multiple candidate statistics as the client timestamp compensation value for unified time correction throughout the entire acquisition phase.

[0046] like Figure 6 As shown, this system acquired three independent sets of cross-host time difference measurement results, where (a), (b), and (c) correspond to the statistical distributions of the first, second, and third sets of data, respectively. According to... Figure 6 The statistical results shown are as follows: the mean of the first group is -18.808 ms, and the median is -21.878 ms; the mean of the second group is -22.458 ms, and the median is -23.334 ms; and the mean of the third group is -17.448 ms, and the median is -21.819 ms. Analysis of the time difference results reveals that each group has a large, unstable single deviation. Therefore, selecting the median of each group as the offset result is more statistically significant.

[0047] Based on the cross-host time difference synchronization results of the three sets of data obtained from the figure, the average value of the first group is -18.808ms, and the median is -21.878ms; the average value of the second group is -22.458ms, and the median is -23.334ms; and the average value of the third group is -17.448ms, and the median is -21.819ms. Analysis of the time difference results shows that each group has unstable single large deviations. Therefore, selecting the median of each group as the offset result is more statistically significant.

[0048] After selecting the fixed offset, this system further evaluates the ability of the compensation strategy to support cross-modal time pairing. Based on the high-density to low-density alignment method, the difference between the timestamps of 10 frames of LiDAR data and 30 frames of camera data is selected as a specific time threshold, using the timestamps of 10 frames of LiDAR data as anchor points. As an alignment method for the same set of time-aligned data, the maximum time difference between the four camera data and the closest LiDAR data is set to less than [a certain value]. To obtain a reasonable time threshold, this system sets multiple candidate time thresholds for the same set of data, statistically compares the sample retention rates under different thresholds, and selects the final matching threshold based on the subsequent dataset construction requirements.

[0049] Specifically, different statistics Under certain conditions, the sample size, sample retention rate, and corresponding minimum time difference distribution characteristics are retained. By observing the trend of the impact of threshold changes on sample size, the study analyzes which threshold can maintain good temporal consistency while ensuring the sample size. If the threshold continues to increase but the rate of increase in sample retention rate has slowed significantly, it indicates that the benefits of further relaxing time constraints are limited, and may instead introduce more paired samples with large time deviations.

[0050] Table 3. Statistical table of sample retention rates under different time thresholds.

[0051] The table shows that the retention rate of the first set of data is only 77.12% at the same 50ms threshold, and only 6.78% when the threshold is set to 15ms. This is because, firstly, the first set of data has the smallest amount of data, so the number of unstable frames at the beginning caused by software starting hardware changes more significantly when the number is the same across all three sets of data. Figure 7 As shown, Figure 7 This diagram illustrates the sample retention rate at different time thresholds. Figure 7 (a) Group 1; (b) Group 2; (c) Group 3; Observation Figure 7 In (a), it was found that there were some timestamp differences greater than 130ms in this set of data. This is because, on the one hand, the first set of data was collected first, and since the equipment was not warmed up sufficiently, there is still room for improvement in equipment stability.

[0052] After understanding the data retention rate at different time thresholds, this system further conducts time difference statistical analysis on the valid samples retained under different time thresholds to evaluate the overall time consistency of the final matching results.

[0053] Figures 8 to 11 The statistical results of the time difference of effective samples under four time thresholds of 15ms, 20ms, 30ms and 50ms are presented respectively. Figure 8 Subgraphs (a), (b), and (c) in the text Figure 9 (d)(e)(f) subgraphs Figure 10 (g)(h)(i) subgraphs Figure 11The (j)(k)(l) subgraphs in the image correspond to the first, second, and third groups of data, respectively; from Figures 8 to 11 It can be seen that when the threshold is less than or equal to 30ms, the maximum duration of the data is small and the distribution is relatively concentrated. This indicates that after fixed offset compensation and threshold screening, the retained samples can correspond to the human motion state at similar times relatively stably. However, considering that the error of 30ms is large for fast-moving human bodies, while the 20ms misalignment usually does not cause obvious visual ghosting or ghosting, although the 15ms threshold is more uniform, it was observed that the data did not show a concentrated distribution at 15ms, while the data performance at 20ms was better in terms of both retention rate and concentration.

[0054] Based on this, the system selects a time threshold of 20 milliseconds, which balances pairing accuracy and sample size, as the final matching threshold for subsequent cross-modal sample construction.

[0055] In addition to numerical statistics, this system also verifies the matching effectiveness through visualization results of cross-modal samples. Specifically, several representative point cloud-image pairing results are selected from the final retained valid samples, and the consistency between the point cloud contours and the main human body regions in the images at key locations such as the torso and limbs is observed. If time compensation and threshold filtering are effective, the main body of the point cloud should be able to fall well near the corresponding human body region in the image, and the overall contour should have a good spatial correspondence. Therefore, the visualization results can provide a supplementary verification of the temporal matching quality from an intuitive perspective.

[0056] like Figure 12 As shown, the images captured by the four cameras at the same time after time synchronization demonstrate different perspectives of the same human movement; for example... Figure 13 As shown, the lidar point cloud data at the same moment maintained a high degree of consistency with the human posture in the image. Through... Figure 12 and Figure 13 It can be observed that after time synchronization, the four cameras display the same action at the same moment. The data from the LiDAR and the cameras are all results from the same moment, maintaining a high degree of consistency in the action.

[0057] S4, Dataset Generation Method for Human Surface Reconstruction Tasks: To construct training samples suitable for human surface reconstruction, these samples need to be further unified with LiDAR observations at the single-frame sample level to form standardized input-supervision pairs that can be used for training the human surface reconstruction network. Unlike constructing only a single form of training input, this system does not directly fix a specific point cloud preprocessing method during the sample generation stage. Instead, it designs four different dataset versions based on the construction of local human point cloud inputs to analyze the impact of different preprocessing strategies on subsequent human surface reconstruction training. Specifically, this system uses human region cropping results as a common foundation, introducing a RANSAC-based ground removal strategy and a DBSCAN-based clustering cleanup strategy, and further constructing a joint cleanup version to ultimately form four types of training datasets: crop, plane, cluster, and clean.

[0058] Let the t-th training sample be denoted as Its basic components can be expressed as (1-32) in, This represents the human point cloud input in frame t. This represents the set of SMPL-related results corresponding to this frame, mainly including information such as pose parameters, shape parameters, model joints, and human surface vertices. This indicates auxiliary labels and statistical information related to the sample.

[0059] Since scene-level LiDAR point clouds typically contain extensive background information, such as the ground, surrounding environmental structures, and discrete noise points, directly feeding them into the network would not only significantly increase the learning difficulty but also weaken the ability to represent the geometric features of the human body. Therefore, this system first constructs a locally cropped input around the spatial region where the human body is located, using it as the common basis for all dataset versions.

[0060] To extract local human body regions from raw LiDAR point clouds, this system designs a human body point cloud extraction process based on spatial constraints and cluster analysis. First, the raw point cloud is initially screened based on the scene's spatial extent, removing irrelevant points that clearly exceed the human activity area. Then, the point cloud is clustered and segmented, and the clustering results are used to filter for human body candidates, incorporating geometric constraints such as point count, target size, and ground clearance. Finally, a 3D bounding region is constructed for the target point cloud, and appropriately expanded to form the local human body point cloud samples required for network input. Let the raw LiDAR point cloud in frame t be... The local constraint region constructed based on the spatial distribution of the human body is The extracted human candidate point cloud can then be represented as: (1-33) This result corresponds to the crop dataset version designed in this system. The crop version only retains the local point cloud near the human body without additional ground removal or clustering purification. Its main function is to extract the local input directly related to the human body surface reconstruction task from the large-scale scene point cloud and serve as the common starting point for all other versions.

[0061] Building upon cropping, this system further introduces two independent point cloud cleanup strategies, corresponding to the plane and cluster dataset versions, respectively. The first strategy is a RANSAC-based ground removal method. Since human point clouds are typically located above the ground, and local cropping results may still retain a large number of ground points, further suppression is necessary. RANSAC (Random Sample Consensus) is a classic robust model fitting method. Its core idea is to repeatedly estimate model parameters from data containing outliers by randomly sampling the smallest subset, and then select the optimal model based on the size of the consensus set. This method is particularly suitable for plane estimation under noisy conditions. For locally cropped point clouds... This system adopts a general planar form: (1-34) Describe the candidate ground model, where Let be the plane normal vector, and d be the plane offset parameter. For any point... Its distance to the plane can be expressed as: (1-35) When satisfied When this point is considered an interior point of the current planar model, then... This represents the distance threshold from a point to a plane. RANSAC selects the plane with the most interior points as the primary ground plane through multiple random samplings and model evaluations, and removes the interior points of this plane from the point cloud. To ensure successful sampling of all interior points at a given confidence level, the theoretical number of iterations can be written as: (1-36) in, Let w be the expected success probability, w be the estimated inlier proportion, and s be the minimum number of sampling points required for a single model fit. The resulting dataset version after this processing is denoted as plane, which is essentially a crop dataset with added ground plane removal. RANSAC is suitable for the main ground suppression task in this system scenario due to its good robustness to outliers.

[0062] The second type of strategy is a clustering purification method based on DBSCAN, such as... Figure 14As shown. Although the crop version has significantly reduced background interference through local cropping, local scattered points, reflection noise, and small-scale background point clusters not connected to the human body may still remain around the human body. To suppress such noise, this system uses DBSCAN (Density-Based Spatial Clustering of Applications with Noise) to perform density clustering analysis on the local point cloud. DBSCAN uses neighborhood radius... The two parameters, `minPts` (min of the minimum number of neighboring points), define the local density structure, enabling the discovery of clusters of arbitrary shapes without pre-specifying the number of clusters, and identifying isolated points in low-density regions as noise. Let the points... of The neighborhood is: (1-37) If satisfied (1-38) Then it is called The core points are defined by density. Based on this, DBSCAN progressively expands clustering through density reachability and density connectivity, and considers points that neither belong to the neighborhood of any core point nor can form an effective cluster as noise points. This system performs DBSCAN clustering on top of the crop version, prioritizing the retention of the main clustering results corresponding to the human subject, thus obtaining the cluster dataset version. This version emphasizes filtering out discrete background points and isolated noise points through local density structure to improve the compactness and purity of the point cloud input.

[0063] Based on the two parallel strategies mentioned above, this system further constructs a combined cleanup version, clean. For example... Figure 15 As shown, specifically, the clean version combines RANSAC-based ground removal and DBSCAN-based clustering purification on top of the crop approach. This means that while preserving the main human body region, it simultaneously suppresses ground redundancy and local outlier noise, resulting in the input version with the highest degree of purification. Compared to plane and cluster, clean emphasizes the complementarity of the two preprocessing strategies: the former mainly targets large-scale principal plane suppression, while the latter mainly targets unstructured background scattering and local noise removal. Through this joint design, the system ultimately obtained four dataset versions with different input purity and background preservation levels, as shown in the figure.

[0064] After constructing the four data versions, this system also needs to standardize the human point cloud according to the network input requirements. Since the number of human point clouds in different frames is usually inconsistent, and subsequent deep learning model training generally requires a fixed number of points as input, this system further resamples or pads the point cloud to ensure all samples have a consistent input scale. Simultaneously, to enhance the geometric comparability between different samples, this system transforms the human point cloud and the corresponding SMPL-related results to a local reference coordinate system. Let the local reference point be... Then the localized human point cloud can be represented as: (1-39) in, This represents the original point coordinates in the human body point cloud. This represents the coordinates of the localized point. Correspondingly, the SMPL joint and human surface vertex results are also translated using the same reference point to ensure that the input point cloud and the supervision results remain consistent under the same local reference frame.

[0065] In addition to the main input and supervision information, this system also constructs auxiliary labels and statistical fields for the samples according to training needs, such as point-level segmentation labels, sample point count information, and sample state markers. This information can be used for auxiliary supervision during the training process and also facilitates sample quality control and data index management. Through the above process, this system ultimately generates four different versions of human point cloud training samples: crop, plane, cluster, and clean.

[0066] like Figure 16 As shown, starting from the original scene-level point cloud (a), a crop version (b) is obtained after cropping the human body region. Based on the crop, RANSAC ground removal is performed to obtain a plane version (c), DBSCAN clustering purification is performed to obtain a cluster version (d), and a clean version (e) is obtained by performing both purification strategies simultaneously. To more intuitively illustrate the impact of different preprocessing strategies on the input quality of the human body point cloud, Figure 17The results further compare the local details of the two dataset versions, plane and clean, with (a) showing the plane result and (b) showing the clean result. It can be seen that the plane version, based on crop, removes the main ground planes using RANSAC, effectively reducing the interference of large-area ground points on the human point cloud. However, it still retains a certain number of discrete background points and local noise points near the human body. In contrast, the clean version further introduces DBSCAN clustering purification after ground removal, filtering out residual small-scale background clusters and isolated outliers. Therefore, its point cloud distribution is more concentrated in the main human body area. From the perspective of local contours, the clean version can more clearly preserve the main structure of the human torso and limbs, while reducing interference from irrelevant points around the human body, demonstrating higher input purity.

[0067] To more intuitively illustrate the impact of different preprocessing strategies on the input quality of human point clouds, the figure presents a comparison of local details between the plane and clean datasets. It can be seen that the plane version, based on cropping, removes the main ground planes using RANSAC, effectively reducing the interference of large-area ground points on the human point cloud. However, it still retains a certain number of discrete background points and local noise points near the human body. In contrast, the clean version further introduces DBSCAN clustering purification after ground removal, filtering out residual small-scale background clusters and isolated outliers. Therefore, its point cloud distribution is more concentrated in the main human body region. From the perspective of local contours, the clean version can more clearly preserve the main structure of the human torso and limbs, while reducing interference from irrelevant points around the human body, demonstrating higher input purity.

[0068] The system constructed four different versions of the dataset based on the human point cloud input: crop, plane, cluster, and clean. Since these versions are all obtained from the same frame of original LiDAR point cloud through region cropping, ground removal, or cluster purification, the differences between the different versions of the point cloud mainly lie in the range of the retained point set and the purity of the point cloud, while their coordinate representation remains unchanged in the original LiDAR coordinate system.

[0069] Therefore, before using it for subsequent network training, it is necessary to verify whether the SMPL results obtained based on EasyMocap can be correctly mapped to the LiDAR coordinate system through existing camera parameters and cross-sensor extrinsic relationships, and whether they maintain reasonable spatial consistency with the human point clouds after four preprocessing strategies. It is also necessary to check whether the point cloud results after different preprocessing strategies still retain the main structure of the human body and whether they can form a correct spatial correspondence with the human surface results.

[0070] Therefore, we first utilize the extrinsic parameter relationship between the camera and LiDAR, involving both internal and external parameters, to uniformly map the SMPL results to the LiDAR coordinate system. Let the SMPL vertex in the camera coordinate system be... Its representation in the LiDAR coordinate system is: (1-40) in, and These represent the rotation matrix and translation vector from the camera to the LiDAR, respectively. The same transformation method can be used for the SMPL joints to obtain smpl_joints_lidar and smpl_verts_lidar.

[0071] Consistency checks were performed visually. Since "clean" represents point cloud data containing all processing relationships, it was only necessary to overlay the SMPL mesh and the clean point cloud of the same frame and observe their spatial alignment. The checks focused on the following three aspects.

[0072] First, is the overall positioning consistent? The main body of the human point cloud should be distributed around the SMPL grid, and should not deviate from it as a whole. If there is obvious translational misalignment, it usually indicates that there is a problem with the time correspondence, the use of extrinsic parameters, or the point cloud clipping range of that frame.

[0073] Secondly, is the orientation of the human body consistent? The torso and limb extension directions of the main body in the point cloud should be consistent with the SMPL mesh. If there is a significant mismatch between the upper and lower limb directions, it indicates an anomaly in the preprocessing or correspondence of the current sample.

[0074] Third, whether the main human body is completely preserved. This is especially crucial for all four versions. Crop mainly checks whether the cropping area covers the entire human body; Plane mainly checks whether the point cloud near the lower limbs and feet is mistakenly deleted after removing the ground; Cluster mainly checks whether the sparse limbs in the local area are misjudged as noise after clustering and purification; Clean checks whether the complete main human body structure is still preserved while the point cloud purity is improved after the two types of processing are superimposed.

[0075] Most of the inspection results showed that the preprocessed human point cloud and the multi-view camera images maintained good mapping consistency.

[0076] like Figure 18As shown, after projecting the preprocessed point cloud onto the corresponding viewpoint image based on the calibration results, it can be observed that the main body of the point cloud and the human body region in the image basically correspond in overall position and contour range, without obvious overall offset or directional error. This indicates that the point cloud cropping, ground removal, and clustering purification processes did not disrupt the geometric correspondence between the original LiDAR point cloud and the camera. Simultaneously, the preprocessed human body point cloud also exhibits good spatial mapping consistency with the corresponding SMPL ground truth, such as... Figure 19 As shown, the main body of the human point cloud can fit well with the main structural regions such as the torso and limbs of the SMPL mesh. The overall positional relationship, posture direction and contour distribution are relatively reasonable. The above results show that the different point cloud versions constructed by this system can still form a stable correspondence with the camera observation information and SMPL supervision results while maintaining the purity of the input, thus meeting the requirements of data consistency for subsequent training of the human surface reconstruction network.

[0077] However, a small portion of the samples were found to be invalid. Among these invalid samples, two main types of issues were identified. First, poor clustering performance led to failure in human point cloud extraction. For example... Figure 20 As shown in the figure, in this type of sample, the clustering results failed to correctly preserve the complete human body. Instead, the main body region was truncated, local limb point clouds were missing, or neighboring background point clusters were incorrectly preserved. This results in the preprocessed point cloud no longer accurately representing the real human body geometrically. Secondly, there is a spatial inconsistency between the SMPL ground truth and the LiDAR point cloud. For example... Figure 21 As shown in the figure, although the main body of the point cloud is still visible in this type of sample, the human point cloud and the corresponding SMPL mesh cannot correspond correctly in certain local positions or pose directions. This manifests as local region offset, inconsistent limb directions, or separation of the point cloud from the model contour, as shown in the figure. Both of these situations weaken the correspondence between the input point cloud and the supervision results. Therefore, they need to be identified as invalid samples and removed during the data processing stage.

[0078] S5, Training Sample Packaging and Dataset Organization. After constructing multiple versions of human point clouds and checking their mapping relationship with camera observations and SMPL results, the next step is to further package all valid data into a standardized dataset that can be directly used for training and testing the human surface reconstruction network. The purpose of this step is to organize the different versions of human point clouds obtained from previous processing, the SMPL-related results output by EasyMocap, and auxiliary labels into training samples with a clear structure, well-defined fields, and easy indexing, thereby providing a stable data foundation for subsequent network training.

[0079] Based on the training requirements of this system, each sample consists of three parts: human point cloud input, SMPL-related supervision results, and auxiliary information. The human point cloud input corresponds to one of four versions: crop, plane, cluster, or clean. The SMPL-related supervision results mainly include pose parameters, shape parameters, model joints, and human surface vertices. The auxiliary information includes sample number, point count, label information, and sample state markers. Let the t-th valid sample be denoted as... Then it can be expressed as: (1-41) in, This represents the human point cloud input in frame t under the k-th point cloud version. This indicates the SMPL parameter results for the corresponding frame; and These represent SMPL joints and SMPL vertices, respectively. This unified representation includes auxiliary labels and statistical information. Through this unified representation, a single sample simultaneously contains network input and corresponding supervision, thus meeting the training requirements for human surface reconstruction tasks.

[0080] In the specific encapsulation process, this system organizes different versions of human point clouds using a unified sample indexing method. For the same frame of data, the four point cloud versions share the same set of SMPL supervision results, with the only difference being the preprocessing method of the input point cloud. Therefore, this system maintains consistent sample numbers in data organization, distinguishing only the input point cloud version. This ensures a strict one-to-one correspondence between the four datasets (crop, plane, cluster, and clean), and also facilitates comparative analysis of different point cloud versions in subsequent training experiments.

[0081] To ensure that subsequent training code can be directly called, this system further saves each sample as a data file with a fixed structure, as shown in the table. For point cloud input, its coordinate information and corresponding auxiliary labels are saved; for SMPL results, supervision fields such as pose parameters, shape parameters, model joints, and human surface vertices are saved; for sample supplementary information, attributes such as the data group to which the current sample belongs, point cloud version, point count, and whether it is a valid sample are recorded. In this way, data from different sources and in different formats are uniformly organized into standardized training samples, thereby reducing the data reading complexity in subsequent training stages.

[0082] Table 4. Description of key fields in training samples

[0083] Regarding sample validity control, samples with poor clustering results, incomplete human subject preservation, or significant anomalies in the spatial mapping relationship between the preprocessed point cloud and the SMPL results are considered invalid and excluded from the formal training, validation, and test sets. Correspondingly, only samples that perform normally in point cloud construction, camera mapping checks, and SMPL mapping checks are retained as valid data for subsequent training and evaluation. This approach reduces the interference of low-quality samples on model training and improves the stability and reliability of subsequent experimental results.

[0084] To ensure the standardization of end-to-end human surface reconstruction model training, parameter selection, and final performance evaluation, this system further divides the constructed valid samples into three parts: a training set, a validation set, and a test set. The training set is used for network parameter learning and model optimization; the validation set is used for model performance monitoring during training, optimal weight selection, and comparison of different training strategies; and the test set is used only for closed evaluation after the model and experimental scheme are finalized, to reflect the model's generalization ability on unseen data as objectively as possible.

[0085] Considering the significant temporal continuity of the data constructed in this study, and the strong correlation between adjacent frames within the same action sequence in terms of human pose, point cloud distribution, and background structure, directly partitioning the dataset randomly at the frame level could easily lead to adjacent samples within the same sequence being distributed across different subsets, resulting in data leakage and affecting the reliability of experimental conclusions. Therefore, this system adopts a dataset construction strategy based on sequence partitioning, that is, all samples in the same acquisition sequence are allocated as a whole to a portion of the training set, validation set, or test set, avoiding the information leakage problem caused by adjacent frames being distributed across sets.

[0086] In the specific partitioning process, this system further considered the imbalance in sample size among different action categories, adopting a partitioning method based on action stratification and weighted by the number of sequence frames. Specifically, firstly, all valid sequences are grouped according to action category. Then, within each action category, weighted allocation is performed based on the number of sample frames contained in the sequence, ensuring that each subset maintains as much consistency as possible in terms of overall sample size and action distribution. Regarding the partitioning ratio, this system divides the valid data into training, validation, and test sets in an 8:1:1 ratio, i.e., the training set accounts for approximately 80%, the validation set approximately 10%, and the test set approximately 10%, as shown in the table below. This partitioning method ensures a sufficient number of training samples while also reserving relatively independent data support for model tuning and final evaluation, resulting in the statistical results shown in the table below.

[0087] Table 5 Dataset Sequence Partitioning

[0088] For the four point cloud versions—crop, plane, cluster, and clean—this system maintains a consistent data group partitioning strategy, meaning that the same data group belongs to the same training or test set across different versions. This ensures that the training results between different versions are strictly comparable, allowing subsequent experimental analyses of the impact of point cloud preprocessing strategies to be based on a unified data partitioning foundation.

[0089] Through the aforementioned training sample encapsulation and dataset organization processes, this system ultimately constructs a standardized dataset for human surface reconstruction tasks. This dataset not only includes human point cloud inputs under different preprocessing strategies but also contains strictly corresponding SMPL parameters, joint and vertex results, and auxiliary label information, thus forming a complete input-supervision pair. Unlike simply adapting the format of raw, self-collected point clouds, this system further refines the data organization process through multi-version point cloud design, mapping relationship checking, invalid sample removal, and data group partitioning, making the final dataset more suitable for subsequent human surface reconstruction network training and experimental analysis.

[0090] Table 6. Statistics on the Division of Training Set, Validation Set, and Test Set

[0091] As shown in the table, the three-part partitioning of the dataset achieved an overall distribution pattern with the training set as the primary component and the validation and test sets as secondary components. On average, the training, validation, and test sets accounted for 76.62%, 9.45%, and 13.93% of the effective data, respectively, with effective data accounting for 95.65% of the total data. This indicates that the vast majority of samples were retained after the initial data cleaning, providing a good data foundation for subsequent model training and evaluation. Looking at individual actions, the shaking, turning, and squatting actions achieved a relatively complete training-validation-test partitioning, while stepping, moving, and waving were all included in the training set. Walking data was only partitioned into the training and validation sets and not included in the test set. This phenomenon is mainly related to the limited number of sequences corresponding to each action and the fact that this system uses a sequence-based partitioning approach: under the premise of ensuring that the same sequence is not split and avoiding leakage of temporally adjacent samples, it is difficult to simultaneously achieve a balanced distribution among the training, validation, and test parts for some actions. Although there are some differences in the division ratios between different actions, the training set still covers all major action types overall, and the validation and test sets also play a role in performance evaluation on some representative actions, which can meet the needs of model training and data generation in the human body surface reconstruction experiment of this system.

[0092] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for constructing a visually assisted lidar human body surface reconstruction dataset, characterized in that, Includes the following steps: Step 1: Construct a laser-visual collaborative acquisition system, which includes a laser radar and a multi-view camera; acquire 3D point clouds of the human body and environment through the laser radar, and acquire human body images from different perspectives through the multi-view camera; record and store the point clouds, the images and their respective timestamps in a unified manner. Step 2: Spatial calibration is performed on the lidar and the multi-view camera, and a cross-sensor spatial relationship is established using a calibration auxiliary device; the pose of the calibration auxiliary device in the camera coordinate system is solved using the multi-view camera, and the pose of the calibration auxiliary device in the lidar coordinate system is solved using the lidar; based on the pose correspondence between the calibration auxiliary device in the camera coordinate system and the lidar coordinate system at the same moment, the extrinsic parameters of the lidar to the camera are solved; Step 3: Time synchronization and frame matching are performed between the lidar and the multi-view camera. The image timestamps of the multi-view camera are compensated and corrected with the time reference of the lidar as a reference. The point cloud frames of the lidar are used as a reference to match the image frames that are closest in time to each point cloud in the compensated image sequence, and effective point cloud-image sample pairs are selected by time threshold. Step 4: Construct multi-source supervision information and human body point cloud samples, and organize the lidar point cloud, the multi-view camera image and the human body surface supervision information at the same moment after spatiotemporal alignment into the same sample level; The original point cloud is subjected to human body region cropping, ground removal based on random sampling consensus algorithm, density-based spatial clustering purification, and combined purification of ground removal and clustering purification, resulting in four human body point cloud input versions with different purification levels: cropped version, ground removal version, clustered version, and combined purification version. The human body surface supervision information is uniformly converted to the lidar coordinate system, and spatial consistency is checked with each version of the human body point cloud input to remove invalid samples. Step 5: Package the training samples and organize the dataset. Package the valid samples into standardized training samples. The standardized training samples include human point clouds, pose parameters, shape parameters, 3D joints, surface vertices and auxiliary labels. Divide the dataset into training set, validation set and test set according to the acquisition sequence to form a dataset for training the human surface reconstruction model.

2. The method of claim 1, wherein the method further comprises: In step 2, the calibration auxiliary device is a checkerboard calibration board; the step of solving the pose of the calibration auxiliary device in the camera coordinate system using the multi-view camera includes: using camera intrinsic parameters, detecting the image corner points of the checkerboard calibration board, and combining the known checkerboard spatial coordinates to solve the rotation matrix and translation vector of the checkerboard calibration board relative to the camera coordinate system; the step of solving the pose of the calibration auxiliary device in the lidar coordinate system using the lidar includes: extracting the point cloud corresponding to the surface of the checkerboard calibration board from the original point cloud, performing plane fitting on the extracted point cloud to obtain the plane normal vector and plane offset, and recovering the pose of the checkerboard calibration board in the lidar coordinate system based on the geometric center of the board surface point cloud and the plane normal vector.

3. The method of claim 2, wherein the method further comprises: In step 2, observation data of the checkerboard calibration board under multiple different postures are collected, and a joint optimization model is constructed to solve the extrinsic parameters of the lidar to the camera. The goal of the joint optimization model is to keep the pose transformation relationship of the checkerboard calibration board in the camera coordinate system and the lidar coordinate system consistent in each set of observation data.

4. The method of claim 1, wherein the method further comprises: In step 3, the compensation and correction of the image timestamps of the multi-view camera specifically includes: using the host connected to the lidar as the time reference end and the host connected to the multi-view camera as the end to be compensated; measuring the time difference between the two hosts multiple times before acquisition, statistically analyzing the multiple measurement results, and selecting the median or truncated mean as a fixed time offset; using the fixed time offset to uniformly compensate the image timestamps of the multi-view camera; setting a time threshold, comparing the time difference between the point cloud frame and its matched image frame with the time threshold, retaining the point cloud-image pair as a valid sample pair when the time difference is less than or equal to the time threshold, and discarding the pair when the time difference is greater than the time threshold; the time threshold is selected based on the statistical results of the sample retention rate under different threshold values.

5. The method for constructing a visually assisted lidar human surface reconstruction dataset as described in claim 4, characterized in that, The time threshold is determined based on the clock synchronization accuracy of the acquisition system, the speed of human movement, and the sample retention rate. When the acquisition frame rate of the lidar is 10Hz and the acquisition frame rate of the multi-view camera is 30Hz, the value of the time threshold ranges from 15 milliseconds to 50 milliseconds.

6. The method for constructing a visually assisted lidar human surface reconstruction dataset as described in claim 1, characterized in that, In step 4, the human body surface supervision information is generated in the following way: using the multi-view camera image after time synchronization, the three-dimensional joints of the human body are recovered through two-dimensional joint detection and multi-view geometric constraints. The parameterized human body model is fitted using the three-dimensional joints of the human body to generate posture parameters, shape parameters, and surface vertices; the generated posture parameters, shape parameters, and surface vertices are then transformed from the camera coordinate system to the lidar coordinate system using the extrinsic parameters obtained in step 2; the parameterized human body model is an SMPL model.

7. The method for constructing a visually assisted lidar human surface reconstruction dataset as described in claim 1, characterized in that, In step 4, the spatial consistency check includes: superimposing the human body point cloud input of each version with the surface vertices transformed into the lidar coordinate system in the same coordinate system to check the overall position consistency and posture direction consistency; and marking samples with incomplete human body after clustering, or with local regional offsets or inconsistent limb directions between the human body point cloud input and the surface vertices as invalid samples and removing them.

8. The method for constructing a visually assisted lidar human surface reconstruction dataset as described in claim 1, characterized in that, In step 5, the step of dividing the training set, validation set, and test set according to the overall acquisition sequence includes: dividing the data by acquisition sequence to ensure that all samples of the same acquisition sequence belong to the same data subset; stratifying the data according to different action categories and weighting the allocation within each action category according to the number of sequence frames; and dividing all valid sequences into the training set, validation set, and test set according to a preset ratio of eight to one to one.

9. A system for constructing a dataset for vision-assisted lidar human body surface reconstruction, characterized in that, The system for performing the method as described in any one of claims 1 to 8 includes: The laser-visual collaborative acquisition unit includes a laser radar, a multi-view camera, a data acquisition and storage terminal, and a calibration auxiliary device. It is used to acquire three-dimensional point clouds of the human body and the environment, human body images from different perspectives, and to uniformly record and store the point clouds, the images, and their timestamps. A spatial calibration unit is used to establish cross-sensor spatial relationships using the calibration auxiliary device and to solve for the extrinsic parameters of the laser radar to the camera. The time synchronization and frame matching unit is used to compensate and correct the image timestamps of the multi-view camera with the time reference of the lidar, and to filter valid point cloud-image sample pairs with the point cloud frames of the lidar as a reference through a time threshold. The data generation and sample construction unit is used to organize the spatiotemporally aligned LiDAR point cloud, the multi-view camera image, and the human surface supervision information into the same sample level. It performs human region cropping, ground removal based on random sampling consensus algorithm, density-based spatial clustering purification, and combined purification of ground removal and clustering purification on the original point cloud to form four human point cloud input versions with different purification levels: cropped version, ground removal version, clustered version, and combined purification version. It also converts the human surface supervision information to the LiDAR coordinate system and performs spatial consistency checks to remove invalid samples. The dataset organization unit is used to encapsulate valid samples into standardized training samples and divide them into training, validation, and test sets according to the collection sequence to form a dataset for training human surface reconstruction models.

10. The system for constructing a vision-assisted lidar human surface reconstruction dataset as described in claim 9, characterized in that, In the data generation and sample construction unit, the human body surface supervision information is generated in the following way: using the multi-view camera image after time synchronization, the three-dimensional joints of the human body are recovered through two-dimensional joint detection and multi-view geometric constraints. The SMPL model is fitted using the three-dimensional human joints to generate posture parameters, shape parameters, and surface vertices; the generated posture parameters, shape parameters, and surface vertices are then transformed from the camera coordinate system to the lidar coordinate system using the extrinsic parameters obtained by the spatial calibration unit.

Images