A multi-person pose estimation method based on millimeter wave radar

Abstract

The application discloses a multi-person pose estimation method based on a millimeter wave radar, and specifically comprises the following steps: acquiring original multi-person point cloud data and real-time joint data, calibrating the two, so that they are in the same three-dimensional space coordinate system, obtaining three-dimensional coordinates of the real-time joint and taking the three-dimensional coordinates as label data; processing the original multi-person point cloud data by using a density-based clustering algorithm, removing noise points and separating the point cloud, obtaining separated multi-person point cloud data for constructing a data set; after preprocessing independent point cloud data to be processed, inputting the preprocessed independent point cloud data into a trained point cloud-based pose estimation neural network to obtain a three-dimensional coordinate prediction value of the real-time joint; the pose estimation neural network is sequentially cascaded by a space embedding layer, a first feature extraction layer, a second feature extraction layer, a global feature extraction layer and a multi-branch full connection layer, and the pose estimation neural network is trained on the data set.

Description

Technical Field

[0001] This invention belongs to the field of attitude estimation, and in particular relates to a multi-person attitude estimation method based on millimeter-wave radar. Background Technology

[0002] Attitude estimation, or posture estimation, refers to the task of inferring the position of human joints using sensors, and plays a crucial role in fields such as health monitoring and motion assessment. Existing attitude estimation algorithms primarily use cameras as sensors, but this raises privacy concerns. Inertial sensor-based methods estimate attitude using sensors worn by the user, but suffer from poor comfort. Millimeter-wave radar, which actively emits millimeter waves for sensing, offers advantages in attitude estimation due to its lack of need for illumination and privacy protection. Existing millimeter-wave radar-based attitude estimation algorithms are mainly divided into single-person attitude estimation algorithms using millimeter-wave point clouds and multi-person attitude estimation algorithms using raw radio frequency signals. Millimeter-wave point clouds offer the advantage of being lightweight, but their sparsity presents challenges for attitude estimation in multi-person scenarios. Raw radio frequency signals contain complete spatial information, but this data type has a large file size, placing a significant burden on storage, transmission, and computation. Summary of the Invention

[0003] The purpose of this invention is to solve the problems existing in the prior art and to provide a multi-person pose estimation method based on millimeter-wave radar. This method uses point clouds collected by millimeter-wave radar as input data, applies a clustering algorithm to separate the point clouds of each person, and uses a neural network to estimate human pose. It is suitable for scenarios with privacy protection requirements.

[0004] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0005] A multi-person attitude estimation method based on millimeter-wave radar includes the following steps:

[0006] S1. Acquire raw multi-person point cloud data and real-time joint data, calibrate the raw multi-person point cloud data and real-time joint data so that the raw multi-person point cloud data and real-time joint data are in the same three-dimensional spatial coordinate system, obtain the three-dimensional coordinates of the real-time joints, and use the three-dimensional coordinates of the real-time joints as label data.

[0007] S2. Use density-based clustering algorithm to process the original multi-person point cloud data, remove noise points from the original multi-person point cloud data and separate the point cloud of each person to obtain the separated multi-person point cloud data. The separated multi-person point cloud data contains the independent point cloud data of each person. The dataset is constructed from the independent point cloud data of each person and the label data.

[0008] S3. Preprocess the independent point cloud data to be processed to obtain preprocessed independent point cloud data. Input the preprocessed independent point cloud data into a trained point cloud-based pose estimation neural network to obtain real-time 3D coordinate prediction values ​​of joints. The pose estimation neural network is composed of a spatial embedding layer, a first feature extraction layer, a second feature extraction layer, a global feature extraction layer, and a multi-branch fully connected layer, which are cascaded in sequence. The pose estimation neural network is trained on the dataset.

[0009] Based on the above scheme, each step can be implemented in the following preferred manner.

[0010] Preferably, in step S1, when acquiring raw multi-person point cloud data and real-time key point data, millimeter-wave radar and Kinect sensor devices are pre-deployed, with the millimeter-wave radar acquiring the raw multi-person point cloud data and the Kinect sensor device acquiring the real-time key point data.

[0011] Preferably, in step S1, when calibrating the original multi-person point cloud data and the real-time key point data, the original multi-person point cloud data is used as a reference, and the real-time key point data is transformed into the three-dimensional spatial coordinate system of the original multi-person point cloud data by coordinate translation. The distance of coordinate translation is equal to the spatial distance between the millimeter-wave radar and the Kinect sensor device.

[0012] Preferably, in step S1, there are a total of 21 real-time joint data points, including: base of spine, middle of spine, neck, head, left shoulder, left elbow, left wrist, left hand, right shoulder, right elbow, right wrist, right hand, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, and center of shoulder.

[0013] Preferably, in step S3, the specific process of preprocessing is as follows: a point cloud sequence is formed from the independent point cloud data to be processed within a preset time period; a sliding window process is performed on the point cloud sequence; the independent point cloud data to be processed within the preset window length range is added together; the addition result is used as the fused point cloud to be processed; the fused point cloud to be processed is filled to a preset size; when filling, the insufficient part is supplemented by the data centers of the independent point cloud to be processed, so as to obtain the preprocessed independent point cloud data.

[0014] Preferably, the specific processing flow in the pose estimation neural network in step S3 is as follows:

[0015] S31. Input the preprocessed independent point cloud data into the spatial embedding layer for spatial embedding to obtain a spatial embedding vector;

[0016] S32. Pass the spatial embedding vector through the first feature extraction layer and the second feature extraction layer in sequence to obtain deep local pose features;

[0017] S33. Input the deep local pose features into the global feature extraction layer to obtain the final global pose features;

[0018] S34. Input the final global pose features into a multi-branch fully connected layer and output the real-time 3D coordinate prediction values ​​of the joints.

[0019] Preferably, in step S3, the specific processing flow in the first feature extraction layer and the second feature extraction layer is the same. In each feature extraction layer, the input data is sampled at the farthest point, and the data points obtained after sampling at the farthest point are used as sampling points. The spatial distance between each sampling point and each data point in the preprocessed independent point cloud data is calculated sequentially in the three-dimensional spatial coordinate system. All data points whose spatial distance calculation results are within a given value are used as the aggregation result. The aggregation result is passed through a feature-sharing multilayer perceptron, and the most significant feature in the aggregation result is selected as the feature of each sampling point through the max pooling operation. The features of all sampling points are used as the features output by the feature extraction layer.

[0020] As a preferred embodiment, the specific processing flow in the global feature extraction layer in step S3 is as follows: input the deep local pose features into the feature-sharing fully connected layer to obtain the initial global pose features, and then perform max pooling on the initial global pose features to obtain the final global pose features.

[0021] Preferably, in step S3, the spatial embedding layer is implemented using a single-layer perceptron.

[0022] Preferably, in step S3, the multi-branch fully connected layer has a total of 21 branches, each branch is used to predict the three-dimensional coordinates of a real-time joint, and the output of each branch is the predicted three-dimensional coordinates of a real-time joint.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] Millimeter-wave radar point clouds serve as a form of sensor data, achieving lightweight transmission and privacy protection. Simultaneously, point cloud-based neural networks are used to predict joint positions, enabling real-time multi-person pose estimation. Attached Figure Description

[0025] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0026] Figure 2 This is a simplified flowchart of the method of the present invention;

[0027] Figure 3 This is a schematic diagram showing the placement of the equipment during the data acquisition process of the method of the present invention;

[0028] Figure 4 This is a structural diagram of the point cloud-based pose estimation neural network of the present invention;

[0029] Figure 5 This is a schematic diagram of the attitude estimation results under five actions according to an embodiment of the present invention. Detailed Implementation

[0030] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0031] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0032] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned multi-person attitude estimation method based on millimeter-wave radar includes the following steps S1 to S3. The specific implementation process of each step will be described in detail below.

[0033] S1. Acquire raw multi-person point cloud data and real-time joint data, calibrate the raw multi-person point cloud data and real-time joint data so that the raw multi-person point cloud data and real-time joint data are in the same three-dimensional spatial coordinate system, obtain the three-dimensional coordinates of the raw multi-person point cloud and the three-dimensional coordinates of the real-time joints, and use the three-dimensional coordinates of the real-time joints as label data.

[0034] It should be noted that in step S1, when acquiring the original multi-person point cloud data and real-time key point data, millimeter-wave radar and Kinect sensor devices are pre-deployed. The millimeter-wave radar acquires the original multi-person point cloud data, and the Kinect sensor device acquires the real-time key point data.

[0035] In this embodiment, a millimeter-wave radar and a Kinect sensor device are first deployed to construct a dataset. The millimeter-wave radar acquires raw multi-person point cloud data and transmits it to a computer via the ROS system. The Kinect sensor device acquires real-time joint point data and transmits it to a computer via the Python-based pykinect library. The computer then performs subsequent data processing, model training, and inference.

[0036] like Figure 3 As shown, a millimeter-wave radar was placed in the scene at a height of 1 meter above the ground. Two subjects stood 2 meters away from the radar, maintaining a minimum distance of 30 centimeters between them, and their range of movement did not exceed 60° of the radar's azimuth angle. A Kinect sensor was placed next to the radar at the same height of 1 meter, ensuring that the subjects were within the Kinect sensor's camera field of view.

[0037] It should be noted that in step S1, when calibrating the original multi-person point cloud data and the real-time joint data, the original multi-person point cloud data is used as a reference, and the real-time joint data is transformed into the three-dimensional spatial coordinate system of the original multi-person point cloud data by coordinate translation. The distance of coordinate translation is equal to the spatial distance between the millimeter-wave radar and the Kinect sensor device.

[0038] In this embodiment, considering that subsequent steps mainly involve point cloud data processing, converting the real-time joint data to the three-dimensional spatial coordinate system of the original multi-person point cloud data through coordinate translation is only a preferred method and can be adjusted according to the actual situation. That is to say, converting the original multi-person point cloud data to the three-dimensional spatial coordinate system of the real-time joint data through coordinate translation is also feasible.

[0039] It should be noted that in step S1 of this invention, there are a total of 21 real-time joint data points, including: base of the spine, middle of the spine, neck, head, left shoulder, left elbow, left wrist, left hand, right shoulder, right elbow, right wrist, right hand, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, and center of the shoulder.

[0040] S2. The original multi-person point cloud data is processed using a density-based clustering algorithm to remove noise points and separate the point cloud of each person to obtain the separated multi-person point cloud data. The separated multi-person point cloud data contains the independent point cloud data of each person. The dataset is constructed from the independent point cloud data of each person and the label data.

[0041] It should be noted that in step S2, the original multi-person point cloud data acquired by the millimeter-wave radar includes human reflection points and noise points. Therefore, it is necessary to separate the point cloud of each person and remove outlier noise points as much as possible. This invention uses a density-based clustering algorithm to process the original multi-person point cloud data. This algorithm can separate the point cloud of any number of people and remove outlier noise points when the distance between people is greater than 30 centimeters.

[0042] like Figure 2 As shown, the original multi-person point cloud data contains point cloud data of many people. After processing by a density-based clustering algorithm, the independent point cloud data belonging to each person in the separated multi-person point cloud data will be clustered together. Figure 2 In the diagram, blue circles represent the independent point cloud data of one person, while bright red circles represent the independent point cloud data of another person.

[0043] It should also be noted that the implementation of the density-based clustering algorithm in step S2 is existing technology and will not be described in detail here.

[0044] S3. Preprocess the independent point cloud data to be processed to obtain preprocessed independent point cloud data. Input the preprocessed independent point cloud data into a trained point cloud-based pose estimation neural network to obtain real-time 3D coordinate prediction values ​​of joints. The pose estimation neural network is composed of a spatial embedding layer, a first feature extraction layer, a second feature extraction layer, a global feature extraction layer, and a multi-branch fully connected layer, which are cascaded in sequence. The pose estimation neural network is trained on the dataset.

[0045] It should be noted that in step S3 of the present invention, the specific preprocessing process is as follows: The specific preprocessing process is as follows: a point cloud sequence is composed of independent point cloud data to be processed within a preset time period; a sliding window processing is performed on the point cloud sequence; the independent point cloud data to be processed within the preset window length range is added together; the addition result is used as the fused point cloud to be processed; the fused point cloud to be processed is filled to a preset size; when filling, the insufficient part is supplemented by the data center points of the independent point cloud to be processed, so as to obtain the preprocessed independent point cloud data.

[0046] It should be noted that, in step S3 of this invention, the specific processing flow in the pose estimation neural network is as follows:

[0047] S31. Input the preprocessed independent point cloud data into the spatial embedding layer for spatial embedding to obtain a spatial embedding vector.

[0048] It should be noted that the above spatial embedding layer is implemented using a single-layer perceptron.

[0049] S32. The spatial embedding vector is passed through the first feature extraction layer and the second feature extraction layer in sequence to obtain deep local pose features.

[0050] It should be noted that the specific processing flow in the first feature extraction layer and the second feature extraction layer is the same. In each feature extraction layer, the input data is sampled at the farthest point. The data points obtained after sampling at the farthest point are used as sampling points. The spatial distance between each sampling point and each data point in the preprocessed independent point cloud data is calculated sequentially in the three-dimensional spatial coordinate system. All data points whose spatial distance calculation results are within a given value are used as the aggregation result. The aggregation result is passed through a feature-sharing multilayer perceptron. The most significant feature in the aggregation result is selected as the feature of each sampling point through the max pooling operation. The features of all sampling points are used as the features output by the feature extraction layer.

[0051] S33. Input the deep local pose features into the global feature extraction layer to obtain the final global pose features.

[0052] It should be noted that the specific processing flow in the global feature extraction layer is as follows: the deep local pose features are input into the feature-sharing fully connected layer to obtain the initial global pose features, and the initial global pose features are subjected to max pooling to obtain the final global pose features.

[0053] S34. Input the final global pose features into a multi-branch fully connected layer and output the real-time 3D coordinate prediction values ​​of the joints.

[0054] It should be noted that the multi-branch fully connected layer has a total of 21 branches. Each branch is used to predict the 3D coordinates of a real-time joint, and the output of each branch is the predicted 3D coordinate value of a real-time joint.

[0055] It should be noted that in this embodiment, a point cloud-based pose estimation neural network is used for pose estimation. During training, the individual point cloud data for each person is first preprocessed to obtain preprocessed point cloud data for each person. The preprocessing method is the same as the preprocessing performed during model inference. Specifically, the individual point cloud data for each person within a preset time period (0.3 seconds) are fused together to obtain a fused point cloud for each person, increasing the number of data points in the point cloud data input to the pose estimation neural network. Since the number of individual point clouds differs for each person, to ensure that the input data size of the pose estimation neural network is the same, the fused point cloud for each person is padded to a preset size. Any insufficient padded portion is supplemented with data points from the unprocessed individual point clouds, resulting in preprocessed point cloud data for each person, containing 128 3D data points.

[0056] like Figure 4As shown, when training the pose estimation neural network, firstly, the spatial embedding layer performs spatial embedding on the preprocessed point cloud data of each person. Each three-dimensional data point shares a single-layer perceptron through a 32-dimensional weight. Since the preprocessed point cloud data of each person contains 128 three-dimensional data points, the spatial embedding vector is 128 32-dimensional features.

[0057] Then, the spatial embedding vector is sequentially passed through multiple stacked feature extraction layers to obtain deep local pose features. Specifically, each feature extraction layer includes farthest point sampling, aggregation, and multilayer perceptron processing. In the first feature extraction layer, farthest point sampling is first performed on the spatial embedding vector, selecting 32 3D data points from 128 3D data points as sampling points, resulting in 32 sampling points, each corresponding to a 32-dimensional feature. The purpose of farthest point sampling is to calculate the spatial distance between each selected sampling point and every 3D data point, maximizing the sum of all distances. All data points whose spatial distance calculation results are within a given value are used as the aggregation result. The aggregation result is then passed through a feature-sharing multilayer perceptron, and the most significant feature is selected through max pooling as the 64-dimensional feature of the farthest point sampling result, which is the shallow local pose feature output by the first feature extraction layer. In the second feature extraction layer, the same operation is performed on the 32 64-dimensional shallow local pose features to obtain 8 128-dimensional deep local pose features.

[0058] In the global feature extraction layer, eight 128-dimensional deep local pose features are taken as input and passed through a feature-sharing fully connected layer to obtain eight initial 128-dimensional global pose features. The most significant feature among all features is selected as the final 128-dimensional global pose feature through max pooling.

[0059] In the multi-branch fully connected layer, a 21-branch fully connected layer is designed. The final global pose features are used as input, and each branch is used to predict the 3D coordinates of a real-time joint. The output of each branch is 3 values, which predict the 3D coordinates of each real-time joint respectively.

[0060] The loss function for the pose estimation neural network is the mean squared error loss function, the batch size is 128, the learning rate is 0.00003, and the training period is 150 epochs. In the above dataset, 80% of the data is used as training data, and 20% is used as test data.

[0061] To better demonstrate the specific implementation and technical effects of the present invention, the multi-person attitude estimation method based on millimeter-wave radar shown in steps S1 to S3 of the above preferred implementation is applied to a specific example below.

[0062] Example

[0063] The implementation process of the multi-person attitude estimation method based on millimeter-wave radar used in this embodiment is as described above and will not be repeated here. The mean absolute error is used as the standard for measuring attitude estimation performance, specifically calculated as the average of the absolute differences between the predicted and actual values ​​of all relevant nodes in the horizontal, depth, and vertical directions.

[0064] Point cloud data was acquired using the TI IWR6843ISK-ODS millimeter-wave radar development board, and keypoint coordinate data was collected by the Kinect V2 sensor for labeling the neural network training. The computation for neural network training and inference was performed on the Tesla T4 GPU.

[0065] The method of the present invention has an average absolute error of less than 6 cm in all three directions, and can achieve inference time within one millisecond with a small number of parameters, thus enabling real-time attitude estimation. The experimental results are shown in Table 1.

[0066] Table 1. Experimental Results

[0067]

[0068] like Figure 5 As shown, the estimation effect of the method of the present invention is good in five action scenarios: stepping in place, waving the left hand, waving the right hand, waving both hands, and alternating waving of the upper arm.

[0069] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for multi-person pose estimation based on millimeter wave radar, characterized in that, The method comprises the following steps: S1. Obtain original multi-person point cloud data and real-time joint data, calibrate the original multi-person point cloud data and the real-time joint data, so that the original multi-person point cloud data and the real-time joint data are in the same three-dimensional space coordinate system, obtain the three-dimensional coordinates of the real-time joint, and take the three-dimensional coordinates of the real-time joint as label data; S2. Process the original multi-person point cloud data by using a density-based clustering algorithm, remove noise points in the original multi-person point cloud data, and separate the point cloud of each person to obtain separated multi-person point cloud data, wherein the separated multi-person point cloud data comprises independent point cloud data of each person, and a data set is constructed from the independent point cloud data of each person and the label data; S3. Preprocess the independent point cloud data to be processed to obtain preprocessed independent point cloud data, input the preprocessed independent point cloud data into a trained point cloud-based pose estimation neural network to obtain a three-dimensional coordinate prediction value of the real-time joint; the pose estimation neural network is sequentially cascaded by a spatial embedding layer, a first feature extraction layer, a second feature extraction layer, a global feature extraction layer, and a multi-branch fully connected layer, and the pose estimation neural network is trained on the data set; In step S1, when the original multi-person point cloud data and the real-time joint data are obtained, the millimeter wave radar and the Kinect sensor device are deployed in advance, the original multi-person point cloud data is obtained by the millimeter wave radar, and the real-time joint data is obtained by the Kinect sensor device; In step S1, when the original multi-person point cloud data and the real-time joint data are calibrated, the original multi-person point cloud data is taken as a reference, and the real-time joint data is converted to the three-dimensional space coordinate system of the original multi-person point cloud data in a coordinate translation manner, and the distance of the coordinate translation is equal to the spatial distance between the millimeter wave radar and the Kinect sensor device; In step S1, the real-time joint data comprises a total of 21 joints, including a spine bottom, a spine middle, a neck, a head, a left shoulder, a left elbow, a left wrist, a left hand, a right shoulder, a right elbow, a right wrist, a right hand, a left hip, a left knee, a left ankle, a left foot, a right hip, a right knee, a right ankle, a right foot, and a shoulder center; In step S3, the specific process of the preprocessing is as follows: a point cloud sequence is formed by the independent point cloud data to be processed within a preset time period, the point cloud sequence is processed by a sliding window, the independent point cloud data to be processed within a preset window length range is added, the addition result is taken as the fusion point cloud to be processed, the fusion point cloud to be processed is filled to a preset size, and the insufficient part is supplemented by using the center point of the independent point cloud data to be processed, and the preprocessed independent point cloud data is obtained; In step S3, the multi-branch fully connected layer has a total of 21 branches, each branch is used to predict a three-dimensional coordinate of a real-time joint, and the output result of each branch is a three-dimensional coordinate prediction value of a real-time joint.

2. The method of claim 1, wherein the method further comprises: In step S3, the specific processing procedure in the pose estimation neural network is as follows: S31. Input the preprocessed independent point cloud data into the spatial embedding layer for spatial embedding to obtain a spatial embedding vector; S32, sequentially input the spatial embedding vector into the first feature extraction layer and the second feature extraction layer, to obtain deep local pose features; S33, input the deep local pose features into the global feature extraction layer, to obtain final global pose features; S34, input the final global pose features into the multi-branch fully connected layer, to output the three-dimensional coordinate prediction value of the real-time joint point.

3. The method of claim 2, wherein the method further comprises: In step S3, the specific processing procedures in the first feature extraction layer and the second feature extraction layer are the same. In each feature extraction layer, the farthest point sampling is performed on the input data, and the data points obtained after the farthest point sampling are taken as sampling points. The spatial distance between each sampling point and each data point in the preprocessed independent point cloud data is sequentially calculated in the three-dimensional coordinate system. All data points with a spatial distance less than a given value are taken as the aggregation result. The aggregation result is input into a feature-shared multi-layer perception, and the most significant feature in the aggregation result is selected as the feature of each sampling point through a max-pooling operation. The features of all sampling points are taken as the features output by the feature extraction layer.

4. The method of claim 3, wherein the method further comprises: In step S3, the specific processing procedure in the global feature extraction layer is as follows: input the deep local pose features into a feature-shared fully connected layer, to obtain initial global pose features. The initial global pose features are input into a max-pooling operation, to obtain final global pose features.

5. The method of claim 1, wherein the method further comprises: In step S3, the spatial embedding layer is implemented by a single-layer perception.

Images