LiDAR-based 3D human motion capture methods, training methods, and systems

Abstract

This invention introduces a system and method for training a machine learning model to generate three-dimensional (3D) motion based on light detection and ranging (LiDAR) point clouds. In various embodiments, a computing system can encode a machine learning model representing objects in a scene. The computing system can train the machine learning model using a dataset comprising synchronous LiDAR point clouds captured by a monocular LiDAR sensor and ground-based 3D motion obtained from an IMU device. The machine learning model can generate 3D motion of objects based on input from multiple point cloud frames captured by the monocular LiDAR sensor.

Description

[0001] Related patent applications

[0002] This application is a continuation application of International Patent Application No. PCT / CN2022 / 078083, filed on February 25, 2022. The foregoing application is incorporated herein by reference in its entirety. Technical Field

[0003] This invention generally relates to motion capture. More specifically, this invention relates to a long-distance three-dimensional human motion capture system and method. Background Technology

[0004] In recent years, deep neural networks have achieved remarkable results in inferring 3D human pose from images or videos. Research on deep neural networks is closely related to the design of datasets, which can be used to train deep neural networks. However, these datasets generally do not include human motion captured at long distances using LiDAR sensors, ground-based human motion acquired by IMU systems, or synchronized color images. Therefore, a training dataset that encompasses depth information and accurate 3D pose ground-based information is needed. Considering the spatiotemporal relationships and temporal order present in point clouds, learning-based methods are typically used to process point clouds. As an alternative to widely used label-based schemes, label-free motion capture technology can reduce the need for body-worn tags in label-based solutions. Summary of the Invention

[0005] This invention introduces a system and method for training a machine learning model to generate three-dimensional (3D) motion based on light detection and ranging (LiDAR) point clouds. In various embodiments, the computing system can encode a machine learning model representing objects in a scene. The computing system can train the machine learning model using a dataset comprising synchronized LiDAR point clouds captured by a monocular LiDAR sensor and ground-based 3D motion obtained from an IMU device. The machine learning model can generate 3D motion of objects based on input from multiple point cloud frames captured by the monocular LiDAR sensor.

[0006] In some embodiments, the object may be the human body.

[0007] In some embodiments, the synchronized LiDAR point cloud may include multiple point cloud frames captured by a monocular LiDAR sensor at a distance from the human body, and each point in the multiple point cloud frames may include a timestamp and an intensity value. The specific distance may range from at least 10 to 50 meters.

[0008] In some embodiments, the ground truth three-dimensional motion can be related to a human body, each ground truth three-dimensional motion of the human body can include a timestamp, a spatial coordinate, and rotations of a plurality of joints of the human body. The ground truth three-dimensional motion can further include a three-dimensional pose of the human body.

[0009] In some embodiments, the dataset can further include synchronized images of the human body.

[0010] In some embodiments, the dataset can include labels of synchronized images of the human body. Each label can include a two-dimensional bounding box around the human body depicted in a synchronized image.

[0011] In some embodiments, the computing system can train a second machine learning model using the synchronized images of the human body in the dataset. The trained second machine learning model can output a second three-dimensional motion of the human body. Based on the second three-dimensional motion of the human body, the three-dimensional motion of the human body can be evaluated.

[0012] In some embodiments, the machine learning model can include a temporal encoder module for extracting global descriptors from each point cloud frame, generating a plurality of hidden variables of the global descriptors, and predicting a plurality of joint positions. In some embodiments, the temporal encoder module can include a PointNet++ network, a bidirectional GRU model, and an MLP encoder. The PointNet++ network can extract the global descriptors, the bidirectional GRU model can generate the plurality of hidden variables, and the MLP encoder can predict the plurality of joint positions.

[0013] In some embodiments, the machine learning model can further include an action solver module for concatenating global features with each joint to generate complete joint features, and for outputting the complete joint features to calculate the plurality of joint rotations. In some embodiments, the action solver module can include an ST-GCN module. The ST-GCN module can learn the concatenated global features through each joint.

[0014] In some embodiments, the machine learning model can further include a joint optimizer module for optimizing the plurality of joint rotations.

[0015] In some embodiments, the joint optimizer module can include an SMPL module. The SMPL module can optimize the plurality of joint rotations.

[0016] The present disclosure introduces a method for generating three-dimensional (3D) motion based on light detection and ranging (LiDAR) point clouds. In various embodiments, a plurality of point cloud frames can be input into a machine learning model. Each point cloud frame can include a plurality of points captured by a monocular LiDAR sensor. The machine learning model can include a temporal encoder module, which includes a feature learning network, a bidirectional GRU, and an MLP encoder. The feature learning network can extract global descriptors for each point cloud frame. The global descriptors can be fed into the bidirectional GRU to generate a plurality of hidden variables. The hidden variables are generally a fusion of temporal information between two or more point cloud frames. The hidden variables can be input into the MLP decoder to predict the positions and rotations of a plurality of joints of an object. The trained machine learning model can output a 3D motion of the object based on the predicted positions and rotations of the plurality of joints.

[0017] In some embodiments, the object can be a human body.

[0018] In some embodiments, the machine learning model can be trained using a dataset including synchronized LiDAR point clouds captured by a monocular LiDAR sensor and ground truth 3D motions obtained from an IMU device.

[0019] In some embodiments, the synchronized LiDAR point clouds can include a plurality of point cloud frames captured by a monocular LiDAR sensor, the monocular LiDAR sensor being at a certain distance from a human body. Each point of the plurality of point cloud frames can include a timestamp and an intensity value. The certain distance can range between at least 10-50 meters.

[0020] In some embodiments, the feature learning network can be a PointNet++ network.

[0021] In some embodiments, the feature learning network can be a Point 4D Transformer.

[0022] In some embodiments, the bidirectional GRU can include a hidden layer for outputting the hidden variables.

[0023] In some embodiments, the positions of the plurality of joints are estimated using the temporal encoder by minimizing a loss, which is expressed by the following equation:

[0024]

[0025] wherein is the predicted joint position of the t-th frame, is the ground truth joint position of the t-th frame.

[0026] In some embodiments, the machine learning model can further include a kinematics solver module. In some embodiments, the kinematics solver can include an ST-GCN module. The ST-GCN model can learn global descriptors and connections to each joint to generate joint features. The joint features can be output to compute the rotations of the plurality of joints.

[0027] In some embodiments, the machine learning model can further include a joint optimizer module. In some embodiments, the joint optimizer can include an SMPL module. The rotations of the plurality of joints can be input into the SMPL module to obtain optimized joint parameters according to which the plurality of joint rotations are optimized.

[0028] In some embodiments, the rotations of the plurality of joints can be estimated by the temporal encoder module, the kinematics solver module, and the joint optimizer module.

[0029] These and other features, and characteristics of the devices, systems, methods and non-transitory computer readable medium disclosed herein, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate similar parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the application. BRIEF DESCRIPTION OF DRAWINGS

[0030] Certain features of various embodiments of the technology are set forth with particularity herein. The features and advantages of the technology will become more apparent from consideration of the following detailed description, taken in conjunction with the accompanying drawings in which:

[0031] Figure 1 An overview of LiDARCap is shown in accordance with various embodiments of the present disclosure.

[0032] Figure 2A A scene in which the LiDARHuman26M dataset can be collected is shown in accordance with various embodiments of the present disclosure.

[0033] Figure 2B A markerless long-range data-driven method for 3D human motion capture using LiDAR point clouds (LiDAR Cap) is shown in accordance with various embodiments of the present disclosure.

[0034] Figure 2C A GRU hidden variable structure is shown in accordance with various embodiments of the present disclosure.

[0035] Figure 3A Results of long distance three-dimensional motion are shown in accordance with various embodiments of the present disclosure.

[0036] Figure 3B A comparison between three-dimensional human motion generated with LiDARCap and three-dimensional human motion generated using state-of-the-art image-based motion capture methods is shown in accordance with various embodiments of the present disclosure.

[0037] Figure 3C A three-dimensional human motion quantitative comparison table is shown in accordance with various embodiments of the present disclosure.

[0038] Figure 3D Qualitative results of three-dimensional human motion are shown in accordance with various embodiments of the present disclosure.

[0039] Figure 4A A computing assembly is shown in accordance with various embodiments of the present disclosure, the computing assembly comprising one or more hardware processors and a machine-readable storage medium storing a set of machine-readable / machine-executable instructions that, when executed, cause the hardware processor(s) to perform a machine learning model training method to generate LiDAR point cloud based three-dimensional motion.

[0040] Figure 4B A computing assembly is shown in accordance with various embodiments of the present disclosure, the computing assembly comprising one or more hardware processors and a machine-readable storage medium storing a set of machine-readable / machine-executable instructions that, when executed, cause the hardware processor(s) to perform a LiDAR point cloud based three-dimensional motion generation method.

[0041] Figure 5 A block diagram of a computer system that can implement various embodiments described herein is shown.

[0042] The accompanying drawings describe various embodiments of the disclosed technology, but are for illustration purposes only, where the same reference numbers are used to identify the same components throughout the drawings. Those skilled in the art will readily recognize, from the following discussion, alternative embodiments of the structures and methods illustrated in the drawings that can be employed without departing from the principles of the disclosed technology described herein. DETAILED DESCRIPTION

[0043] Recently, markerless human motion capture has developed rapidly in applications such as virtual reality, augmented reality, and interactive entertainment. In these applications, conveniently capturing distant 3D human motion in large spaces can be challenging, which is crucial for live events. Conventional methods employ vision-based motion capture (mocap) solutions to capture 3D human motion. High-end solutions in this approach utilize dense optical markers or dense camera track rigging for 3D human motion capture. For consumer use, these solutions are not economically feasible. In contrast, monocular capture methods are more practical and attractive. Furthermore, recent advancements in learning technologies have enabled reliable human motion capture from single video streams. For example, using pre-scanned human templates or parametric human models, human motion can be determined from a single video stream. However, in long-distance shooting scenarios, where performers are far from the camera, images captured using these methods are susceptible to degradation and blur artifacts, making motion capture vulnerable.

[0044] Various 3D human motion capture methods have been explored for scenarios involving degraded and low-resolution image capture. However, these methods remain vulnerable at long distances, failing to capture global position, especially when dealing with untextured clothing or changes in ambient lighting. Therefore, to handle untextured clothing and varying ambient lighting, motion capture using body-worn sensors such as inertial measurement units (IMUs) is widely adopted because it is unaffected by environmental changes. However, requiring performers to wear body-worn sensors makes this particular solution unsuitable for capturing the movements of people in everyday clothing. Furthermore, IMU-based solutions are susceptible to accumulated global drift artifacts, especially at long distances. Since the effective shooting range of cameras (such as RGBD cameras) is relatively short, typically less than 5 meters, alternative solutions using consumer-grade RGBD sensors for human motion capture are also undesirable for long-distance capture in large scenes.

[0045] The invention addresses the aforementioned problems. According to various embodiments of the invention, a consumer-grade LiDAR-based solution is proposed to solve these problems. The LiDAR-based solution may include a LiDAR sensor. The LiDAR sensor can provide accurate depth information for large-scale scenes with a large effective range (up to 30m). Under normal lighting conditions, the solution can capture human motion at long distances without artifact degradation in the camera's (e.g., RGBD camera) visual sensor. However, capturing long-distance 3D human motion using a LiDAR sensor (e.g., a monocular LiDAR sensor) remains challenging. First, at long distances, the effective observation point cloud corresponding to the target performer may be sparse and noisy, making reliable motion capture difficult. Second, while 3D modeling using LiDAR sensors is prevalent, most existing solutions focus on scene understanding and 3D perception. Third, the lack of large-scale LiDAR-based datasets with accurate 3D human motion annotations makes data-driven motion capture pipelines using LiDAR infeasible.

[0046] To address these challenges, according to various embodiments of the present invention, a label-free, long-range data-driven motion capture method using a single LiDAR sensor or LiDAR capture (LiDARCap) is provided, such as... Figure 1 As shown. Figure 1 An overview 100 of LiDARCap according to various embodiments of this disclosure is shown. For example... Figure 1As shown, the LiDARCap overview 100 may include a dataset 102 for training LiDARCap. Once LiDARCap is trained, it can generate 3D human motion of performer 104 based on multiple LiDAR point clouds captured from a LiDAR sensor (such as a monocular LiDAR sensor). In some embodiments, the dataset may be the LiDARHuman26M dataset. In various embodiments, the LiDARHuman26M dataset may be a large benchmark dataset for LiDAR-based human motion capture. The LiDARHuman26M dataset may include various modalities, such as synchronous LiDAR point clouds 102a, visual (e.g., RGB) images 102b, and ground-based live 3D human motion 102c obtained from a professional IMU-based mocap device. In some embodiments, the LiDARHuman26M dataset may include 20 everyday actions, 13 performers with 184.0k capture frames, and approximately 26 million valid 3D points of the observed performers, with a large capture distance range between 12m and 28m. In this regard, the LiDARHuman26M dataset is the first dataset for data-driven LiDAR-based human motion capture in a long-range setting. Furthermore, according to other embodiments of the invention, a strong baseline motion capture method for LiDAR point clouds is provided. Additionally, according to other embodiments of the invention, state-of-the-art image-based methods are used as a baseline for LiDARCap evaluation. These evaluations highlight the advantages of LiDARCap compared to image-based methods in a long-range setting.

[0047] LiDARCap method

[0048] Markerless 3D motion capture in distant scenes can be challenging using conventional methods with cameras. For example, 2D cameras cannot provide depth information, and depth-sensing cameras can only operate at close or short distances. Therefore, LiDAR sensors offer the advantage of both a long sensing range and the ability to acquire depth information. According to various embodiments of the invention, a human motion dataset (e.g., the LiDARHuman26M dataset) containing LiDAR point clouds over distant human motion scenes can be provided alongside synchronized IMU-captured ground footage. Furthermore, an end-to-end model capable of inferring an optimal parametric human model from the LiDAR point cloud can be provided. In some cases, a skinned multi-person linear (SMPL) model can be employed to represent human pose and body shape in a compact manner. In some embodiments, the SMPL model may include pose parameters θ∈R related to human motion. 72In some embodiments, the pose parameters can be formulated as rotations of the 23 joints relative to their parent joints and global body rotation of the root joint. In some embodiments, in addition to the pose parameters, the SMPL model may also separately include body shape parameters β∈R. 10 These parameters can control human height, weight, and limb proportions related to human movement. In some embodiments, in addition to posture and body shape parameters, the SMPL model may also separately include translation parameters t∈R. 3 These parameters can be used when human body position is required. In various embodiments, the SMPL model can deform a 3D mesh associated with human movement. For example, in one particular implementation, the SMPL model can deform a triangulated mesh template with 6890 vertices based on human pose and body shape parameters. In this case, the triangulated mesh can be formulated as follows:

[0049] Training dataset

[0050] In general, long-range motion capture has great potential in various applications, such as immersive VR / AR experiences and motion quality assessment. According to various embodiments of the present invention, the present invention further provides in detail the first long-range LiDAR-based motion capture dataset, namely the LiDARHuman26M dataset. Figure 2A A scenario 200 is illustrated, illustrating the collection of the LiDAR Human26M dataset according to various embodiments of this disclosure. For example... Figure 2A As shown, the LiDARHuman26M dataset can be collected in two separate scenarios. For example, as Figure 2A As shown, the first scene 202 depicts a terrace supporting long-distance human capture. The second scene 204 is an open space between two buildings, supporting large capture tilt angles to avoid self-occlusion. The capture device (i.e....) Figure 2AThe setup details for the LiDAR sensor and camera shown are illustrated in Table 206. In this example, the object (i.e., the human body) in the first scene 202 is located 12-28 meters away from the acquisition device. Furthermore, in this example, the acquisition device is located 5 meters above the object. Similarly, the object in the second scene 204 is 14-24 meters away from the acquisition device, which is located 7 meters above the object. For the data collection in the first scene 202 and the second scene 204, a total of 13 performers were recruited to participate in the data acquisition of the LiDARHuman26M dataset, with each scene lasting from 15 to 30 minutes. As a result of this data collection, the LiDARHuman26M dataset provides 184,048 frames of point cloud data and 26,414,383 data points, and includes animations of 20 different types of daily actions, such as walking, running, and exercising. The LiDARHuman26M dataset comprises three data modalities: synchronized LiDAR point clouds, color (RGB) images, and ground-based live 3D human motion from a professional IMU-based mocap device. The preprocessed LiDARHuman26M dataset removes background from point cloud data frames and localization errors associated with the IMU-based mocap device. In some embodiments, after data collection and preprocessing, the LiDARHuman26M dataset may include synchronized images. These synchronized images may be labeled to include tags. Each tag may include a two-dimensional region or bounding box surrounding an object depicted in the synchronized image. Typically, there may be a one-to-one correspondence between the tag associated with an object and the pose associated with the object. In some embodiments, each data point in the ground-based live 3D motion data 102c may include a timestamp, spatial coordinates, and rotation associated with, or related to, a performer's joint as measured by the IMU device. In some embodiments, the LiDARHuman26M dataset may further include global coordinates of the joints and the rotations associated with them. In some embodiments, synchronized images can be captured using a camera capable of capturing images at 30 frames per second and a resolution of 1920×1080. The camera must capture images in the red, green, and blue channels, and add a timestamp to each image. In some embodiments, a monocular LiDAR capable of capturing point clouds at 10 frames per second can be used to capture point clouds from the LiDARHuman26M dataset, with approximately 70,000 points (e.g., laser points) per point cloud frame. Each point in the point cloud can be described using Cartesian coordinates and intensity values. Finally, a timestamp can be added to each frame of the point cloud.

[0051] Figure 2ATable 208 shows the statistics of the LiDAR Human26M dataset compared with other publicly available 3D human pose datasets. As shown in Table 208, the LiDAR Human26M dataset has the following characteristics: First, the LiDAR Human26M dataset contains many long-distance (up to 28 meters) human movements, while the image datasets in Table 208 (such as Human3.6M, HumanEva, etc.) typically have limited capture distances. Second, the LiDAR Human26M dataset involves a maximum of 20 everyday movements, while the other datasets in Table 208 have very limited movements. Third, the LiDAR Human26M dataset involves three different modalities, including point clouds, RGB video, and mocap ground footage provided by IMU devices. Furthermore, the image-based datasets in Table 208 do not provide depth information, which is crucial for long-distance motion capture.

[0052] LiDARCap

[0053] Figure 2B This invention illustrates a label-free, long-range data-driven method for 3D human motion capture using LiDAR point clouds (LiDAR Cap) according to various embodiments of the present disclosure. Figure 2B The LiDARCap shown can be trained on the LiDARHuman26M dataset. In this type of embodiment, LiDARCap can acquire point cloud sequences from a monocular LiDAR sensor as input and output a 3D human motion sequence. In some embodiments, LiDARCap may include a processing module ( Figure 2B (Not shown in the diagram), a time encoder module 222, an inverse kinematics solver module 224, and an SMPL joint optimizer module 226. Each module will be discussed in further detail below.

[0054] In some embodiments, the preprocessing module can sample data points in the point cloud frame into a specific amount of data. For example, suppose the input LiDAR point cloud sequence P = {P_T} is T frames long. (t) |t=1…T}, and each frame contains an arbitrary number of points P. (t) ={p i (t) |i=1…n t In this example, by performing a uniform downsampling operation through sampling or repetition, the preprocessing module fixes the number of points per frame at 512. Alternatively, the preprocessing module could fix the number of points per frame at 1024. Many variations are possible and can be envisioned.

[0055] In some embodiments, the time encoder module 222 can extract the global descriptor for each point cloud frame. In some embodiments, the time encoder module 222 can be implemented using a PointNet++ network. The PointNet++ network can be used as the backbone to extract the global descriptor for each point cloud frame P. (t) global descriptor f (t) For example, a PointNet++ network can be used to extract a 1024-dim global descriptor. In some embodiments, the time encoder module 222 may further include a bidirectional gated regression unit (bi-GRU). The bi-GRU can be used to combine temporal information and frame-by-frame features. (t) Integrate into the global descriptor to generate the hidden variable g. (t) Dual GRUs are typically a variant of regressive neural networks (RNNs). In this configuration, the RNN can stack the structure of hidden variables and can expand the hidden variables over time, such as... Figure 2C As shown. Figure 2C The structure of the dual GRU hidden variable 240 according to various embodiments of this disclosure is shown. Figure 2C In this configuration, a dual GRU may include at least one hidden layer 242, and the output of the hidden layer is a hidden variable. Each hidden variable is simply a tensor of shape 1×C.

[0056] In some embodiments, the time encoder module 222 may further include a multilayer perceptron (MLP) decoder. The MLP decoder can receive g (t) As input, to predict the corresponding joint position In some embodiments, the loss of the time encoder module 222 It can be represented as:

[0057]

[0058] In the formula, Let t be the ground-based joint position in frame t.

[0059] In some embodiments, the inverse kinematics solver module 224 can extract features from the predicted joints graphically. In some embodiments, the inverse kinematics solver module 224 can be implemented using a spatiotemporal graph convolutional network (ST-GCN). For example, in one embodiment, ST-GCN can be used as the backbone to extract features from the predicted joints graphically. In some embodiments, ST-GCN can concatenate frame-by-frame global features with each joint to generate complete joint features. As graph nodes. Subsequently, the output of ST-GCN can be fed into the regressor ( Figure 2B (Not shown in the image) This is used to calculate joint rotation. When calculating the loss, the 6D rotation can be mapped to the final axis-angle format. Here, the 6D rotation representation is chosen as an intermediate result to obtain better continuity. In some embodiments, the loss of the module... Represented as:

[0060]

[0061] In the formula, Let be the ground pose parameters for frame t.

[0062] In some embodiments, the SMPL joint optimizer module 226 can be configured as the final stage of LiDARCap to further improve the regression of θ. In some embodiments, the SMPL joint optimizer module 226 may include a skinned multi-person linear (SMPL) model. Joint rotations can be fed into the SMPL model to obtain 24 joints on the SMPL mesh. The predicted joints and ground reality can be reused again in the SMPL joint optimizer module 226. The loss is used to improve the accuracy of the regression θ. The difference is that in the first stage, the joints are regressed directly by an MLP-based decoder, while here, the joints are sampled on parameterized mesh vertices determined by θ. In some embodiments, the loss of the SMPL joint optimizer module 226 is... It can be represented as:

[0063]

[0064] in, From the posture parameters Joint positions sampled in a parameterized SMPL mesh. This step typically provides stronger constraints on the regression of θ in a geometrically intuitive way. Disengagement experiments are conducted in this manner to demonstrate its desirability.

[0065] In various embodiments, LiDARCap can be trained by optimizing the following formulated joint loss function L in an end-to-end manner:

[0066]

[0067] LiDARCap training

[0068] In various embodiments, LiDARCap is trained using the Adam optimizer, and the descent rate of the GRU layer and ST-GCN is set to 0.5 for 200 training cycles. Batch normalization layers can then be applied to every convolutional layer except the final output layer before the MLP decoder. In one embodiment, LiDARCap can be trained using a processor such as a graphics processing unit (GPU) or a central processing unit (CPU). The batch size can be set to 8, and the learning rate can be set to 1 × 10⁻⁶. –4 The attenuation rate is 1×10 –4 During the evaluation phase of LiDARCap, the network architecture is trained using the most suitable learning rate until convergence is achieved. In some embodiments, LiDARCap can be trained on the LiDARHuman26M dataset; experimental details are as follows. Figures 3A-3D As shown.

[0069] Experiment

[0070] Generally, the LiDARCap method proposed in this invention can predict human actions in distant scenes very well, such as... Figure 3A As shown. Figure 3A The results of long-distance three-dimensional motion according to various embodiments of this disclosure are shown. Figure 3A Three graphical rows, 302, 304, and 306, are displayed. The first graphical row, 302, describes the performer's image data captured by the camera. The second graphical row, 304, describes the corresponding point cloud data of the performer captured by the LiDAR sensor. The third graphical row, 306, describes the corresponding 3D human motion of the performer generated by LiDAR ARp. Figure 3A As shown, highly accurate 3D human motion can be generated (or constructed) based on point cloud data captured from LiDAR sensors. Figure 3A As shown, LiDARCap can accurately generate 3D human motion when the data points in the point cloud frame are sparse. For example, in Figure 3A In this example, a first 3D human motion 312 is generated based on point cloud data with many data points. However, in this example, a second 3D human motion 314 is generated based on point cloud data with fewer data points. In this example, both the first 3D human motion 312 and the second 3D human motion 314 accurately reflect the human motion description in the corresponding image data. Therefore, the LiDARCap disclosed in this invention improves upon existing methods for capturing 3D human motion at long distances.

[0071] To facilitate further research, the LiDARCap method was compared with state-of-the-art (SOTA) image-based motion capture methods. Figure 3BThis paper illustrates a comparison between 3D human motion generated using LiDARCap according to various embodiments of this disclosure and 3D human motion generated using state-of-the-art (SOTA) image-based motion capture methods. Quantitative and qualitative comparisons are made with HMR and VIBE, the latter also relying on temporal encoding. Figure 3B As shown, thanks to the 3D resolution of LiDAR point clouds, the LiDARCap method outperforms state-of-the-art image-based methods.

[0072] Figure 3C Table 342 illustrates a quantitative comparison of three-dimensional human motion according to various embodiments of this disclosure. Table 342 shows the corresponding quantitative comparisons of three-dimensional human motion using different evaluation metrics. Table 342 reports the Protodyakonov alignment mean joint error (PA-MPJPE), mean (per) joint position error (MPJPE), correct joint ratio (PCK), and vertex mean error (PVE). Error metrics are measured in millimeters. Additionally, acceleration error (m / s²) is also included. 2 ) Recording is an important evaluation metric for sequence data. Thanks to the effective utilization of 3D spatial information, the LiDARCap method significantly outperforms HMR and VIBE.

[0073] Evaluation

[0074] To investigate the impact of different components in the LiDARCap method, two separation experiments were conducted. The first experiment verified the effectiveness of the combination of the PointNet++ time encoder module and ST-GCN. The second experiment verified the effectiveness of the combination of the time encoder module and the inverse kinematics module. Figure 3C Table 344 summarizes the results of these two experiments. In another experiment, the PointNet++ (P++) backbone was replaced with other carefully designed network architectures, such as P4Transformer (P4T), the Attention (ATT) module, and the Voting (VOT) module. To fuse spatiotemporal information, P4Transformer was used instead of the original P4Transformer as the backbone for the latter two. The necessity of leveraging ST-GCN to utilize the temporal dimension rather than the joint features on the dual GRU was also evaluated. Figure 3C Table 346 shows the comparison results above, from which it can be concluded that for LiDARCap, more complex operations (such as attention and voting) will lead to a performance degradation. Global features contribute to achieving optimal performance, and there is no significant difference between P++ and P4T. Furthermore, the introduction of a tree-like kinematic chain can better locate adjacent joints than dual GRU, which only affects frame-by-frame global features. Convolutions at the same joint time step also ensure continuity and consistency in an explicit manner.

[0075] Generally, the LiDARCap method yields good results in long-range motion capture due to the superior properties of point clouds. To verify the generality of the LiDARCap method, it was tested on pedestrian point cloud sequences from the KITTI detection dataset and the Waymo Open Dataset, such as... Figure 3D As shown. Figure 3D Qualitative results of three-dimensional human motion according to various embodiments of this disclosure are shown. (By...) Figure 3D As can be seen, the LiDARCap method can learn the correct gait and global orientation of pedestrians. For clearly defined upper limbs, the LiDARCap method can determine the correct location, while for unclear limbs, LiDARCap can make reasonable guesses based on prior information from the time series.

[0076] Figure 4A A computing component 400 according to various embodiments of the present disclosure is illustrated. The computing component includes one or more hardware processors 402 and a machine-readable storage medium 404 (stored with a set of machine-readable / machine-executable instructions). When executed, the instructions cause the hardware processors 402 to perform a machine learning model training method to generate 3D motion based on LiDAR point clouds. The computing component 400 may be... Figure 5 The computing system 500 in the middle. The hardware processor 402 may include Figure 5 The processor 504 or any other processing unit described in this invention. Machine-readable storage medium 404 may include main memory 506, read-only memory (ROM) 508, etc. Figure 5 The memory 510 and / or any other suitable machine-readable storage medium described in this invention.

[0077] At box 406, a machine learning model representing an object in the scene is encoded. In some embodiments, the object may be a human body or a performer. In some embodiments, the machine learning model may include a temporal encoder module for extracting a global descriptor from each point cloud frame, generating multiple hidden variables of the global descriptor, and predicting multiple joint positions. In some embodiments, the machine learning model may further include a motion solver module for concatenating global features with each joint to generate complete joint features, and for outputting the complete joint features to calculate the rotation of the multiple joints. In some embodiments, the machine learning model may further include a joint optimizer module for optimizing the rotation of the multiple joints.

[0078] At box 408, a machine learning model can be trained using a dataset comprising synchronized LiDAR point clouds captured by a monocular LiDAR sensor and ground-based real-time 3D motion obtained from an IMU device. In some embodiments, the synchronized LiDAR point cloud may include multiple point cloud frames captured by a monocular LiDAR sensor at a specific distance from the human body, and each point in the multiple point cloud frames may include a timestamp. In some embodiments, the specific distance may be at least 10-50 meters from the human body. In some embodiments, the ground-based real-time 3D motion may be associated with the human body, and each ground-based real-time 3D motion of the human body may include a timestamp, spatial coordinates, rotation of multiple joints of the human body, and the 3D pose of the human body. In some embodiments, the dataset may further include synchronized human body images. The synchronized human body images may include labels, and each label may include a 2D region or bounding box containing the human body.

[0079] At box 410, the machine learning model can generate a 3D motion of an object based on input from multiple point cloud frames captured by a monocular LiDAR sensor. In some embodiments, a second machine learning model can be trained using synchronized images from the dataset. The trained second machine learning model can output a second 3D motion. Based on the second 3D motion, the 3D motion can be evaluated.

[0080] Figure 4B A computing component 450 according to various embodiments of the present disclosure is illustrated. The computing component includes one or more hardware processors 452 and a machine-readable storage medium 454 (stored with a set of machine-readable / machine-executable instructions). When executed, the instructions cause the hardware processors 452 to perform a 3D motion generation method based on LiDAR point clouds. The computing component 450 may be... Figure 5 The computing system 500 in the middle. The hardware processor 452 may include Figure 5 The processor 504 or any other processing unit described in this invention. Machine-readable storage medium 454 may include main memory 506, read-only memory (ROM) 508, ... Figure 5 The memory 510 and / or any other suitable machine-readable storage medium described in this invention.

[0081] At box 456, multiple point cloud frames can be input into a machine learning model. Each point cloud frame may include multiple points captured by a monocular LiDAR sensor. In some embodiments, the machine learning model may include a time encoder module comprising a feature learning network, a bidirectional GRU, and an MLP encoder. In some embodiments, the machine learning model can be trained using a dataset comprising synchronized LiDAR point clouds captured by a monocular LiDAR sensor and ground-based 3D motion obtained from an IMU device. In some embodiments, the synchronized LiDAR point cloud may include multiple point cloud frames captured by a monocular LiDAR sensor at a specific distance from the human body, and each point in the multiple point cloud frames includes a timestamp and an intensity value. In some embodiments, the specific distance may be at least 10-50 meters from the human body.

[0082] At box 458, the feature learning network extracts a global descriptor for each point cloud frame. In some embodiments, the feature learning network may be a PointNet++ network. In some embodiments, the feature learning network may be a Point4D Transformer.

[0083] At box 460, a global descriptor can be fed into a bidirectional GRU to generate multiple hidden variables. In some embodiments, the bidirectional GRU may include a hidden layer that can output the hidden variables.

[0084] At box 462, hidden variables can be input to the MLP decoder to predict the position and rotation of multiple joints of an object. In some embodiments, the object can be a human body.

[0085] At frame 464, the trained machine learning model can output the object's 3D motion based on the predicted positions and rotations of the multiple joints.

[0086] For example, the technology described in this invention can be implemented by one or more dedicated computing devices. The dedicated computing device may execute the technology using a hard-wired approach, or may include circuitry or digital electronic equipment, such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) (which can execute the technology after continuous programming), or may include one or more hardware processors (which, after programming, can execute the technology according to program instructions in firmware, memory, other storage devices, or combinations thereof).

[0087] Figure 5This diagram illustrates a computer system 500 that can implement the various embodiments described in this invention. The computer system 500 includes an information transport bus 502 or other communication mechanism, and one or more hardware processors 504 coupled to the bus 502 for information processing. The description of tasks performed by the device is intended to represent tasks performed by the one or more hardware processors 504.

[0088] Computer system 500 also includes a main memory 506, such as random access memory (RAM), cache memory, and / or other dynamic storage devices, coupled to bus 502, for storing information and instructions to be executed by processor 504. During the execution of instructions to be executed by processor 504, main memory 506 can also be used to store temporary variables or other intermediate information. When these instructions are stored in storage media accessible to processor 504, they transform computer system 500 into a customized special-purpose machine for performing the operations specified in the instructions.

[0089] The computer system 500 further includes a read-only memory (ROM) 508 or other static storage device coupled to a bus 502 for storing static information and instructions required by the processor 504. A storage device 510, such as a disk, optical disk, or USB thumb drive (flash drive), coupled to the bus 502, is provided for storing information and instructions.

[0090] Computer system 500 can be coupled to output device 512 (such as a cathode ray tube (CRT) or LCD display (or touch screen)) via bus 502 for displaying information to the computer user. Input device 514, including alphanumeric and other keys, is coupled to bus 502 for transmitting information and command selection to processor 504. Another user input device is cursor controller 516. Computer system 500 also includes a communication interface 518 coupled to bus 502.

[0091] Unless the context otherwise requires, the term “comprise” and its variations, such as “comprises” and “comprising”, shall be interpreted in an open, inclusive sense, meaning “including but not limited to”, throughout this specification and claims. Statements of numerical ranges are intended as shorthand notations to refer individually to each individual value within a range (including the range-defined value), and each individual value is part of this specification as it is individually enumerated in this invention. Furthermore, unless the context explicitly specifies otherwise, the singular forms “a,” “an,” and “the” include plural references. Phrases such as “at least one of,” “choose at least one from a set,” or “choose at least one from a set comprising” require interpretation within the context of disjunctive concepts (e.g., they should not be interpreted as at least one of A and at least one of B).

[0092] The reference to "an embodiment" or "an embodiment" in this specification means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing in various parts of this specification do not necessarily all refer to the same embodiment, but may refer to the same embodiment in some cases. Furthermore, in one or more embodiments, particular features, structures, or characteristics may be combined in any suitable manner.

[0093] A component implemented as another component can be interpreted as a component that operates in the same or similar manner as another component, and / or includes the same or similar features, characteristics, and parameters as another component.

Claims

1. A method for training a laser-radar based three-dimensional human motion capture model, the method comprising the steps of: encoding, by a computing system, a machine learning model representing an object in a scene; training, by the computing system, the machine learning model using a dataset comprising synchronized LiDAR point clouds captured by a monocular LiDAR sensor and ground truth three-dimensional motions obtained from an IMU device; wherein the machine learning model generates a three-dimensional motion of the object based on an input of a plurality of point cloud frames captured by the monocular LiDAR sensor; wherein the machine learning model comprises a temporal encoder module for extracting global descriptors from each point cloud frame, generating a plurality of hidden variables of the global descriptors, and predicting a plurality of joint positions; wherein the temporal encoder module comprises a PointNet++ network, a bi-directional GRU, and an MLP encoder; wherein the PointNet++ network extracts the global descriptors, the bi-directional GRU generates the plurality of hidden variables, and the MLP encoder predicts the plurality of joint positions.

2. The method of claim 1, wherein, the object is a human body.

3. The method of claim 2, wherein, the synchronized LiDAR point clouds comprise a plurality of point cloud frames captured by a monocular LiDAR sensor, the monocular LiDAR sensor being at a distance from the human body, each point of the plurality of point cloud frames comprising at least one timestamp, wherein the distance is in a range of at least 10-50 meters.

4. The method of claim 2, wherein, the ground truth three-dimensional motions are related to the human body, each ground truth three-dimensional motion of the human body comprising a timestamp, a spatial coordinate, a plurality of joint rotations of the human body, and a three-dimensional pose of the human body.

5. The method of claim 2, wherein, the dataset further comprises synchronized images of the human body.

6. The method of claim 5, wherein, the dataset further comprises labels of the synchronized images of the human body, wherein each label comprises a two-dimensional bounding box surrounding the human body depicted in the synchronized image.

7. The method of claim 6, further comprising the steps of training, using the synchronized images of the human body in the dataset, a second machine learning model; outputting, by the trained second machine learning model, a second three-dimensional motion of the human body; evaluating, based on the second three-dimensional motion of the human body, the three-dimensional motion of the human body.

8. The method of claim 7, wherein, the machine learning model further comprises a kinematics solver module for connecting the global features with each joint to generate complete joint features, and for outputting the complete joint features to calculate the plurality of joint rotations, wherein the kinematics solver comprises an ST-GCN for connecting the global features with each joint.

9. The method of claim 8, wherein, the machine learning model further comprises a joint optimizer module for optimizing the rotations of the plurality of joints, wherein the optimizer module comprises an SMPL model for optimizing the rotations of the plurality of joints.

10. A system for training a laser-radar based three-dimensional human motion capture model, the system comprising: a processor; A storage device configured to store a machine learning model representing an object in a scene and a dataset, the dataset comprising synchronized LiDAR point clouds captured by a monocular LiDAR sensor and ground truth three-dimensional motions obtained from an IMU device, wherein the machine learning model is configured to generate three-dimensional motions of the object based on inputs of a plurality of point cloud frames captured by the monocular LiDAR sensor; a memory configured to store instructions that, when executed by the processor, enable the system to train the machine learning model using the dataset; wherein the machine learning model comprises a temporal encoder module configured to extract global descriptors from each point cloud frame, generate a plurality of hidden variables of the global descriptors, and predict a plurality of joint positions; wherein the temporal encoder module comprises a PointNet++ network, a bi-directional GRU, and an MLP encoder; wherein the PointNet++ network is configured to extract the global descriptors, the bi-directional GRU is configured to generate the plurality of hidden variables, and the MLP encoder is configured to predict the plurality of joint positions.

11. The system of claim 10, wherein, The object is a human body.

12. The system of claim 11, wherein, The synchronized LiDAR point clouds comprise a plurality of point cloud frames captured by a monocular LiDAR sensor, the monocular LiDAR sensor is separated from the human body by a distance, each point of the plurality of point cloud frames comprises a timestamp, and the distance is in a range of at least 10-50 meters.

13. The system of claim 11, wherein, The ground truth three-dimensional motions are related to the human body, each ground truth three-dimensional motion of the human body comprises a timestamp, a spatial coordinate, a plurality of joint rotations of the human body, and a three-dimensional pose of the human body.

14. The system of claim 11, wherein, The dataset further comprises synchronized images of the human body.

15. The system of claim 14, wherein, The dataset further comprises labels of the synchronized images of the human body, each label comprises a two-dimensional bounding box surrounding the human body depicted in the synchronized image.

16. The system of claim 15, wherein, When the processor executes the instructions, the instructions enable the system to: train a second machine learning model using the synchronized images of the human body in the dataset; output a second three-dimensional motion of the human body by the trained second machine learning model; evaluate the three-dimensional motion of the human body based on the second three-dimensional motion of the human body.

17. The system of claim 16, wherein, The machine learning model further comprises a kinematics solver module configured to connect the global features with each joint to generate complete joint features, and configured to output the complete joint features to calculate the plurality of joint rotations, wherein the kinematics solver comprises an ST-GCN configured to connect the global features with each joint.

18. The system of claim 17, wherein, The machine learning model further comprises a joint optimizer module configured to optimize the rotations of the plurality of joints, wherein the optimizer module comprises an SMPL model configured to optimize the rotations of the plurality of joints.

19. A method of laser radar-based three-dimensional human motion capture, the method comprising the steps of: inputting a plurality of point cloud frames to a machine learning model, each point cloud frame comprising a plurality of points captured by a monocular LiDAR sensor, the machine learning model comprising a temporal encoder module, the temporal encoder module comprising a feature learning network, a bi-directional GRU, and an MLP encoder; extracting global descriptors of each point cloud frame by the feature learning network; feeding the global descriptors into the bidirectional GRU to generate a plurality of hidden variables; inputting the hidden variables into an MLP decoder to predict positions and rotations of a plurality of joints of the subject; outputting, by the trained machine learning model, a three-dimensional motion of the subject based on the predicted positions and rotations of the plurality of joints.

20. The method of claim 19, wherein, The subject is a human body.

21. The method of claim 19, further comprising the steps of: training the machine learning model using a dataset comprising synchronized LiDAR point clouds captured by a monocular LiDAR sensor and ground truth three-dimensional motions obtained from an IMU device.

22. The method of claim 21, wherein, The synchronized LiDAR point clouds comprise a plurality of point cloud frames captured by a monocular LiDAR sensor at a distance from a human body, each point of the plurality of point cloud frames comprising a timestamp and an intensity value, wherein the distance ranges between at least 10-50 meters.

23. The method of claim 19, wherein, The feature learning network is a PointNet++ network.

24. The method of claim 19, wherein, The feature learning network is a Point 4D Transformer.

25. The method of claim 19, wherein, The bidirectional GRU comprises one hidden layer for outputting the hidden variables.

26. The method of claim 19, further comprising the steps of: estimating, by the temporal encoder, the positions of the plurality of joints by minimizing a loss, represented by the formula: In the formula (t) is a predicted joint position for the tth frame, is a ground truth joint position for the tth frame.

27. The method of claim 19, wherein, The machine learning model further comprises an inverse kinematics solver module, wherein the inverse kinematics solver module comprises an ST-CGN, and the method further comprises the steps of: connecting, by the inverse kinematics solver module, the global descriptors with each joint to generate joint features; outputting the joint features to compute the rotations of the plurality of joints.

28. The method of claim 27, wherein, The machine learning model further comprises a joint optimizer module, wherein the joint optimizer module comprises an SMPL model, and the method further comprises the steps of: inputting the rotations of the plurality of joints into the joint optimizer module to obtain optimized rotations of the plurality of joints.

29. The method of claim 28, further comprising the steps of: estimating, by the temporal encoder module, the rotations of the plurality of joints.

Images