Human body posture estimation method based on millimeter wave radar point cloud

Abstract

The invention discloses a human body posture estimation method based on millimeter wave radar point clouds, which comprises the following steps of: processing each frame of input millimeter wave radar sparse point clouds by utilizing a built posture estimation model, and finally predicting and outputting a three-dimensional coordinate sequence of corresponding human body key joint points; wherein the attitude estimation model is composed of a point cloud completion network based on an encoder-decoder architecture, a global-local double-branch network and a feature fusion and prediction mechanism, and the method comprises the following steps: firstly, enhancing the density and integrity of an original sparse point cloud by using the point cloud completion network; the method comprises the following steps: complementing point clouds and original point clouds, respectively processing the complemented point clouds and original point clouds through a global-local double-branch network, extracting global structure features and local detail features, finally fusing the two features through a feature fusion and prediction mechanism, predicting three-dimensional coordinates of human body joint points through a regression layer, and realizing accurate attitude estimation. The method improves the estimation precision and robustness, maintains the privacy protection advantage, and is suitable for complex application scenes.

Description

Technical Field

[0001] This invention relates to the technical field of human pose estimation, and in particular to a human pose estimation method based on millimeter-wave radar point clouds. Background Technology

[0002] Human pose estimation aims to accurately infer the three-dimensional spatial coordinates of key human joints from various sensor data, thereby reconstructing human pose. Precise human pose information is crucial for a deeper understanding of human behavior, intentions, and states, demonstrating significant application value and enormous potential in cutting-edge fields such as human-computer interaction, intelligent healthcare, autonomous driving, and sports science.

[0003] Traditional human pose estimation techniques primarily rely on optical sensors, such as RGB cameras and depth cameras. While these techniques have made significant progress, their performance is susceptible to environmental factors such as changes in lighting and occlusion. Furthermore, their deployment in privacy-sensitive settings such as homes or medical facilities presents ethical and regulatory challenges.

[0004] In recent years, millimeter-wave radar has emerged as a promising alternative for human posture estimation. Millimeter-wave radar can penetrate non-metallic obstacles and operate reliably in low-light, smoky, and even completely dark environments, demonstrating strong environmental adaptability. More importantly, the radar's output information does not contain identifiable biometric information, effectively protecting user privacy. These characteristics make millimeter-wave radar an ideal choice for all-weather, all-scenario, and privacy-preserving human sensing applications.

[0005] However, achieving high-precision human pose estimation using millimeter-wave radar point clouds faces significant challenges, primarily due to the inherent characteristics of the generated point clouds. Compared to lidar or structured light systems, radar point cloud RPCs generated by commercial millimeter-wave radar typically exhibit significant sparsity and incompleteness. This inherent sparsity makes it difficult for RPCs to adequately depict the fine geometry of the human body. Therefore, directly applying deep learning models designed for dense point clouds often results in limited performance, failing to achieve the desired joint localization accuracy. How to effectively address the sparsity problem of millimeter-wave RPCs and, based on this, achieve robust and accurate human pose estimation is a pressing technical challenge in this field. Summary of the Invention

[0006] The purpose of this invention is to address the problems of low accuracy and poor robustness in existing millimeter-wave radar human pose estimation methods due to the inherent sparsity, incompleteness, and noise interference of radar point clouds. This invention proposes a human pose estimation method based on millimeter-wave radar point clouds, which can significantly improve the accuracy and stability of human pose estimation in various scenarios, while retaining the privacy protection advantages of millimeter-wave radar.

[0007] To achieve the above objectives, the technical solution provided by this invention is as follows: a human pose estimation method based on millimeter-wave radar point clouds. This method utilizes a constructed pose estimation model to process each frame of sparse point cloud from millimeter-wave radar, ultimately predicting and outputting the corresponding three-dimensional coordinate sequence of key human joints, thereby achieving accurate estimation of human pose. The pose estimation model consists of a point cloud completion network based on an encoder-decoder architecture, a global-local dual-branch network, and a feature fusion and prediction mechanism. The specific implementation of this method includes the following steps:

[0008] 1) The sparse point cloud of each frame of millimeter-wave radar is processed by the point cloud completion network to obtain the completed point cloud that can better capture the overall structure of the human body, that is, increase the point cloud density to make it closer to the real shape of the human body, and provide a more reliable input data basis for subsequent pose estimation.

[0009] 2) The spatiotemporal dynamic changes of human posture are learned from both global and local levels through a global-local dual-branch network. This fully utilizes the global information of the completed point cloud and the local information of the original millimeter-wave radar sparse point cloud to provide richer features for the final posture prediction. The global-local dual-branch network includes a global branch and a local branch. The global branch receives the completed point cloud as input and extracts the features of the overall human structure through a convolutional neural network (CNN) to grasp the basic outline of the human posture. The local branch processes the original millimeter-wave radar sparse point cloud, focusing on extracting local motion and reflection details to capture subtle changes in various parts of the human body. The features from both branches are then input into a long short-term memory network (LSTM) to extract dynamic characteristics over time.

[0010] 3) The different features extracted from the global and local branches are integrated through the feature fusion and prediction mechanism. The mechanism first concatenates the global and local features to form a fusion feature vector containing complete information. Then, the fusion feature vector is processed by the multilayer perceptron (MLP) and finally the three-dimensional coordinate sequence of the key joints of the human body is predicted by regression, thereby completing the estimation of human posture.

[0011] Furthermore, the point cloud completion network performs the following operations on each frame of the input millimeter-wave radar sparse point cloud:

[0012] The encoder of the point cloud completion network first processes each frame of millimeter-wave radar sparse point cloud input through a multilayer perceptron (MLP) to obtain point features containing local geometric information. The MLP independently extracts high-dimensional features for each point in the sparse point cloud, and its weights are shared among all points, thereby effectively learning the local geometric information of the point cloud and forming an initial feature representation for each point. Subsequently, the point features are input into multiple multi-head self-attention blocks. By calculating the dependencies between point features, the global structural information inside the point cloud is captured, and downsampling is performed in combination with farthest point sampling. Finally, a global feature vector that condenses global context information is output. The multi-head self-attention block generates attention weights by calculating the query, key, and value similarity between point features, and weights and aggregates the features of all points, thereby effectively capturing long-distance dependencies and global structural information inside the point cloud.

[0013] The decoder of the point cloud completion network first receives a global feature vector and a set of seed points obtained by performing farthest point sampling on the input sparse point cloud or the initially generated coarse point cloud. Then, it performs multiple upsampling operations. In each upsampling stage, the global feature vector is fused with the features of each seed point or the points generated in the previous stage, and a new set of point coordinates is predicted and generated, thereby gradually increasing the point cloud density and refining the point cloud structure, and finally generating a completed point cloud with the target density.

[0014] Furthermore, the point cloud completion network generates a denser and more complete point cloud based on a global feature vector and a set of initial seed points. The output completed point cloud contains N... completed The seed points are derived from a subset of the input sparse point cloud. During the upsampling process, the global feature vector is combined with the features of each seed point through multiple multilayer perceptrons, and more new 3D point coordinates are gradually generated. This process not only increases the number of point clouds, but also optimizes the position of the newly added points, so that the final output completed point cloud can more accurately reflect the real surface morphology of the human body.

[0015] Furthermore, the encoder will convert a single frame of millimeter-wave radar sparse point cloud P sparse ∈R B×N×3As input, R is the set of real numbers, B is the batch size, N is the maximum number of points per frame, and 3 represents the xyz 3D coordinates. First, the encoder uses a Multilayer Perceptron (MLP) to perform preliminary feature extraction on each point, mapping the original 3D coordinates to a 64-dimensional feature space and applying the ReLU activation function. Then, multi-head self-attention blocks are used to process the 64-dimensional features to capture long-range dependencies and global structural information within the point cloud. After one multi-head self-attention block, the number of points is reduced to N / 2 using farthest-point sampling. Next, the feature dimension is expanded from 64 to 128 using the MLP, and multi-head self-attention blocks are applied again to enhance global information interaction, outputting 128-dimensional features. Then, farthest-point sampling is used again to downsample the number of points to N / 4. After each self-attention block, residual connections are added and layer normalization is applied to stabilize the effect. Finally, global pooling is used to aggregate all point features output from the second farthest-point sampling, forming a 128-dimensional global feature vector Z. global .

[0016] Furthermore, the decoder is based on the global feature vector Z. global To gradually generate a complete point cloud, firstly, a multilayer perceptron (MLP) generates the global feature vector Z. global The vector is mapped to an M×3 dimensional vector and reshaped into a (M,3) low-density point cloud, providing a coarse structural basis, where M represents the number of points in the low-density point cloud. These newly generated points are compared with the original sparse point cloud P. sparse Merge the points, and then use farthest point sampling to select a representative subset from the merged set of points, called the seed point P. seed Next, independent multilayer perceptrons (MLPs) process the seed point and Z-axis respectively. global The point feature matrix F is obtained. seed and global shape feature F shape , will F shape Copy to match F seed The number of rows, and with F seed The fused feature matrix F is formed by concatenating features along the feature dimensions. concat Then, the number of points is gradually increased through a multi-stage upsampling process. At each stage, spatial interpolation is used to generate new points and restore more details. In each upsampling stage, a multilayer perceptron (MLP) is used for feature refinement, and ReLU activation function and dropout layer are used to prevent overfitting. Finally, the decoder uses a multilayer perceptron (MLP) to map the obtained high-dimensional point features back to 3D coordinate space to generate the final completed point cloud P. completed .

[0017] Furthermore, when training the point cloud completion network, the point cloud P is minimized. completed With the target point cloud P gt The Chamfe distance loss function LCD To optimize network parameters, the loss function is defined as:

[0018]

[0019] In the formula, p represents a point in the completed point cloud, and q represents a point in the target point cloud.

[0020] Furthermore, the global-local dual-branch network processes different forms of point cloud data streams in parallel through global and local branches, so as to comprehensively utilize the structural information of the completed point cloud and the detailed information of the original sparse point cloud, as follows:

[0021] The global branch receives the completed point cloud P generated by the point cloud completion network. completed As input; the local branch receives the original millimeter-wave radar sparse point cloud P. sparse As input, each frame of the millimeter-wave radar sparse point cloud contains the three-dimensional coordinates, Doppler velocity, and reflection intensity information of each point. Both branches adopt a cascaded structure. First, a convolutional neural network (CNN) with shared weights independently extracts the spatial features of each frame's point cloud, forming a temporally ordered sequence of frame-level spatial features. Then, these temporally ordered sequence of frame-level spatial features are input into their respective long short-term memory (LSTM) networks to learn and capture the dynamic information of human posture changes over time. Finally, the global branch outputs the final hidden state H, which summarizes the overall spatiotemporal features of the structure. global_final The local branch output summarizes the final hidden state H, which encapsulates the local details and spatiotemporal features of motion. local_final .

[0022] Furthermore, the completed point cloud P completed And the original millimeter-wave radar sparse point cloud P sparse Before entering their respective branches, frame-level preprocessing is performed: the point cloud is an unordered collection. To adapt to convolution operations, the point cloud in each frame is first sorted in descending order according to its x, y, and z coordinates, and then reshaped into a matrix format in row-major order to make it suitable for CNN processing. After preprocessing, the input tensor shape of the global branch is (T,G,G,3), and the input tensor shape of the local branch is (T,L,L,5), where T is the time step, G is the side length of the global branch input matrix, L is the side length of the local branch input matrix, (T,G,G,3) means that the xyz coordinates of each frame are reshaped into a G×G matrix, and (T,L,L,5) means that the xyz coordinates, Doppler velocity, and reflection intensity information are encoded using an L×L×5 tensor.

[0023] Each branch employs a cascaded convolutional neural network (CNN) and long short-term memory (LSTM) architecture for spatiotemporal feature extraction. A weight-shared CNN independently processes each frame of point cloud data. The CNN consists of several 3×3 convolutional layers, each followed by batch normalization, ReLU activation, and a 20% dropout layer. Subsequently, a global average pooling layer compresses the obtained frame-level spatial features into a 128-dimensional vector feature, which summarizes the spatial information of each frame. The temporally ordered sequence of frame-level spatial features is input into the respective LSTMs. The LSTMs learn the dynamic information of pose changes over time and understand how the pose evolves. Each LSTM outputs its final 128-dimensional hidden state, which encodes the comprehensive spatiotemporal context features of the input sequence of its corresponding branch.

[0024] Furthermore, the feature fusion and prediction mechanism will output the final hidden state H from the global branch. global_final The final hidden state H of the local branch output local_final By concatenating along the feature dimension, a fused feature vector F containing two complementary pieces of information is generated. fused :

[0025] F fused =Concat(H global_final H local_final )

[0026] Subsequently, the feature vector F will be fused. fused The input is fed into a prediction head, which is a multilayer perceptron (MLP) consisting of two fully connected layers. This prediction head directly regresses the total number of N cells in the human body. J A sequence of coordinates of predefined key points in three-dimensional space in This represents the three-dimensional coordinates of the i-th predicted joint.

[0027] Furthermore, the training process of the pose estimation model aims to minimize the predicted key coordinate sequence J. pred Corresponding ground real key point coordinate sequence Mean square error loss L between MSE , where (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th real joint; the mean square error loss L MSE Defined as:

[0028]

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] 1. Significantly improves point cloud quality and attitude estimation accuracy: Through an innovative point cloud completion network, the sparsity and incompleteness of millimeter-wave radar point clouds are effectively overcome, providing high-quality input for subsequent attitude estimation; combined with a global-local dual-branch network, the structural information of the completed point cloud and the detailed information of the original millimeter-wave radar sparse point cloud are fully utilized and effectively fused, significantly improving the accuracy of human attitude estimation.

[0031] 2. Enhanced robustness of attitude estimation: The proposed attitude estimation model can better cope with the challenges brought by radar data sparsity, noise and complex scenes, making the attitude estimation results more stable and reliable.

[0032] 3. Methodological innovation and practicality: The attitude estimation model proposed in this invention, especially the idea of ​​point cloud completion and global-local information fusion, provides a new and effective way to solve the challenges of millimeter-wave radar in the field of human perception, and provides new ideas for other downstream tasks based on millimeter-wave radar point clouds. It has good academic value and potential application prospects. Attached Figure Description

[0033] Figure 1 This is an overall flowchart of the method of the present invention.

[0034] Figure 2 Complete the network architecture diagram for point clouds.

[0035] Figure 3 This is an architecture diagram of a global-local dual-branch network. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings.

[0037] like Figures 1 to 3 As shown, this embodiment discloses a human pose estimation method based on millimeter-wave radar point clouds. This method utilizes a constructed pose estimation model to process each frame of sparse millimeter-wave radar point cloud input, ultimately predicting and outputting the corresponding three-dimensional coordinate sequence of key human joints, thereby achieving accurate estimation of human pose. The pose estimation model consists of a point cloud completion network based on an encoder-decoder architecture, a global-local dual-branch network, and a feature fusion and prediction mechanism. The specific implementation of this method includes the following steps:

[0038] 1) The sparse point cloud of each frame of millimeter-wave radar is processed by the point cloud completion network to obtain the completed point cloud that can better capture the overall structure of the human body, that is, increase the point cloud density to make it closer to the real shape of the human body, and provide a more reliable input data basis for subsequent pose estimation.

[0039] 2) The spatiotemporal dynamic changes of human posture are learned from both global and local levels through a global-local dual-branch network. This fully utilizes the global information of the completed point cloud and the local information of the original millimeter-wave radar sparse point cloud to provide richer features for the final posture prediction. The global-local dual-branch network includes a global branch and a local branch. The global branch receives the completed point cloud as input and extracts the features of the overall human structure through a convolutional neural network (CNN) to grasp the basic outline of the human posture. The local branch processes the original millimeter-wave radar sparse point cloud, focusing on extracting local motion and reflection details to capture subtle changes in various parts of the human body. The features from both branches are then input into a long short-term memory network (LSTM) to extract dynamic characteristics over time.

[0040] 3) The different features extracted from the global and local branches are integrated through the feature fusion and prediction mechanism. The mechanism first concatenates the global and local features to form a fusion feature vector containing complete information. Then, the fusion feature vector is processed by the multilayer perceptron (MLP) and finally the three-dimensional coordinate sequence of the key joints of the human body is predicted by regression, thereby completing the estimation of human posture.

[0041] Specifically, the point cloud completion network performs the following operations on each frame of the input millimeter-wave radar sparse point cloud:

[0042] The encoder of the point cloud completion network first processes each frame of millimeter-wave radar sparse point cloud input through a multilayer perceptron (MLP) to obtain point features containing local geometric information. High-dimensional features are extracted independently for each point in the sparse point cloud, and the weights of the MLP are shared among all points, thus effectively learning the local geometric information of the point cloud and forming an initial feature representation for each point. Subsequently, the point features are input into multiple multi-head self-attention blocks. By calculating the dependencies between point features, the global structural information within the point cloud is captured, and downsampling is performed using farthest point sampling. Finally, a global feature vector condensing global context information is output. The multi-head self-attention block generates attention weights by calculating the query, key, and value similarity between point features, and weights and aggregates the features of all points, thereby effectively capturing long-distance dependencies and global structural information within the point cloud.

[0043] The decoder of the point cloud completion network first receives a global feature vector and a set of seed points obtained by performing farthest point sampling on the input sparse point cloud or the initially generated coarse point cloud. Then, it performs multiple upsampling operations. In each upsampling stage, the global feature vector is fused with the features of each seed point or the points generated in the previous stage, and a new set of point coordinates is predicted and generated, thereby gradually increasing the point cloud density and refining the point cloud structure, and finally generating a completed point cloud with the target density.

[0044] Specifically, the point cloud completion network generates a denser and more complete point cloud based on a global feature vector and a set of initial seed points. The output completed point cloud contains N... completed A point, for example, N. completed =196; The seed points are derived from a subset of the input sparse point cloud. During the upsampling process, the global feature vector is combined with the features of each seed point through multiple multilayer perceptrons, and more new three-dimensional point coordinates are gradually generated. This process not only increases the number of point clouds, but also optimizes the position of the newly added points, so that the final output completed point cloud can more accurately reflect the real surface morphology of the human body.

[0045] In a preferred embodiment of this example, see Figure 2 As shown, the encoder will convert a single frame of millimeter-wave radar sparse point cloud P sparse ∈R B×N×3 As input, R is the set of real numbers, B is the batch size, N is the maximum number of points per frame, and 3 represents the xyz 3D coordinates. First, the encoder uses a Multilayer Perceptron (MLP) to perform preliminary feature extraction on each point, mapping the original 3D coordinates to a 64-dimensional feature space and applying the ReLU activation function. Then, multi-head self-attention blocks are used to process the 64-dimensional features to capture long-range dependencies and global structural information within the point cloud. After one multi-head self-attention block, the number of points is reduced to N / 2 using farthest-point sampling. Next, the feature dimension is expanded from 64 to 128 using the MLP, and multi-head self-attention blocks are applied again to enhance global information interaction, outputting 128-dimensional features. Then, farthest-point sampling is used again to downsample the number of points to N / 4. After each self-attention block, residual connections are added and layer normalization is applied to stabilize the effect. Finally, global pooling is used to aggregate all point features output from the second farthest-point sampling, forming a 128-dimensional global feature vector Z. global .

[0046] In a preferred embodiment of this example, see Figure 2 As shown, the decoder is based on the global feature vector Z. global To gradually generate a complete point cloud, firstly, a multilayer perceptron (MLP) generates the global feature vector Z. global The points are mapped to an M×3 dimensional vector and reshaped into a (M,3) low-density point cloud, providing a coarse structural basis, where M represents the number of points in the low-density point cloud, the value of which depends on the specific task. These newly generated points are compared with the original sparse point cloud P. sparse Merge the points, and then use farthest point sampling to select a representative subset from the merged set of points, called the seed point P. seed Next, independent multilayer perceptrons (MLPs) process the seed point and Z-axis respectively. global The point feature matrix F is obtained.seed and global shape feature F shape , will F shape Copy to match F seed The number of rows, and with F seed The fused feature matrix F is formed by concatenating features along the feature dimensions. concat Then, the number of points is gradually increased through a multi-stage upsampling process. At each stage, spatial interpolation is used to generate new points and restore more details. In each upsampling stage, a multilayer perceptron (MLP) is used for feature refinement, and ReLU activation function and dropout layer are used to prevent overfitting. Finally, the decoder uses a multilayer perceptron (MLP) to map the obtained high-dimensional point features back to 3D coordinate space to generate the final completed point cloud P. completed .

[0047] When training the point cloud completion network, minimize the completed point cloud P. completed With the target point cloud P gt The Chamfe distance loss function L CD To optimize network parameters, the loss function is defined as:

[0048]

[0049] In the formula, p represents a point in the completed point cloud, and q represents a point in the target point cloud.

[0050] Specifically, the global-local dual-branch network processes different forms of point cloud data streams in parallel through global and local branches, in order to comprehensively utilize the structural information of the completed point cloud and the detailed information of the original sparse point cloud, see [link to relevant documentation]. Figure 3 As shown, the details are as follows:

[0051] The global branch receives the completed point cloud P generated by the point cloud completion network. completed As input; the local branch receives the original millimeter-wave radar sparse point cloud P. sparse As input, each frame of the millimeter-wave radar sparse point cloud contains the three-dimensional coordinates, Doppler velocity, and reflection intensity information of each point. Both branches adopt a cascaded structure. First, a convolutional neural network (CNN) with shared weights independently extracts the spatial features of each frame's point cloud, forming a temporally ordered sequence of frame-level spatial features. Then, these temporally ordered sequence of frame-level spatial features are input into their respective long short-term memory (LSTM) networks to learn and capture the dynamic information of human posture changes over time. Finally, the global branch outputs the final hidden state H, which summarizes the overall spatiotemporal features of the structure. global_final The local branch output summarizes the final hidden state H, which encapsulates the local details and spatiotemporal features of motion. local_final .

[0052] In a preferred embodiment of this example, the point cloud completion P completed And the original millimeter-wave radar sparse point cloud P sparse Before entering their respective branches, frame-level preprocessing is performed: Point clouds are unordered collections. To adapt to convolution operations, the point clouds in each frame are first sorted in descending order according to their x, y, and z coordinates, and then reshaped into a matrix format in row-major order to make it suitable for CNN processing. After preprocessing, the input tensor shape of the global branch is (T, 14, 14, 3), and the input tensor shape of the local branch is (T, 8, 8, 5), where T is the time step, for example, T = 4. (T, 14, 14, 3) means that the xyz coordinates of each frame are reshaped into a 14×14 matrix, and the 8×8×5 tensor encodes the xyz coordinates, Doppler velocity, and reflection intensity information.

[0053] Each branch employs a cascaded convolutional neural network (CNN) and long short-term memory (LSTM) architecture for spatiotemporal feature extraction. A weight-shared CNN independently processes each frame of point cloud data. The CNN consists of several 3×3 convolutional layers, each followed by batch normalization, ReLU activation, and a 20% dropout layer. Subsequently, a global average pooling layer compresses the obtained frame-level spatial features into a 128-dimensional vector feature, which summarizes the spatial information of each frame. The temporally ordered sequence of frame-level spatial features is input into the respective LSTMs. The LSTMs learn the dynamic information of pose changes over time and understand how the pose evolves. Each LSTM outputs its final 128-dimensional hidden state, which encodes the comprehensive spatiotemporal context features of the input sequence of its corresponding branch.

[0054] Specifically, the feature fusion and prediction mechanism will output the final hidden state H from the global branch. global_final The final hidden state H of the local branch output local_final By concatenating along the feature dimension, a fused feature vector F containing two complementary pieces of information is generated. fused :

[0055] F fused =Concat(H global_final H local_final )

[0056] Subsequently, the feature vector F will be fused. fused The input is fed into a prediction head, which is a multilayer perceptron (MLP) consisting of two fully connected layers. This prediction head directly regresses the total number of N cells in the human body. J A sequence of coordinates of predefined key points in three-dimensional space in This represents the three-dimensional coordinates of the i-th predicted joint.

[0057] To optimize the parameters of the entire pose estimation model, the training process of the pose estimation model aims to minimize the predicted keypoint coordinate sequence J. pred Corresponding ground real key point coordinate sequence Mean square error loss L between MSE , where (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th real joint; the mean square error loss L MSE Defined as:

[0058]

[0059] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Therefore, any changes made in accordance with the shape and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for human pose estimation based on millimeter-wave radar point clouds, characterized in that, This method utilizes a constructed pose estimation model to process the sparse point cloud of each frame of millimeter-wave radar input, ultimately predicting the corresponding 3D coordinate sequence of key human joints, thereby achieving accurate estimation of human pose. The pose estimation model consists of a point cloud completion network based on an encoder-decoder architecture, a global-local dual-branch network, and a feature fusion and prediction mechanism. The specific implementation of this method includes the following steps: 1) The sparse point cloud of each frame of millimeter-wave radar is processed by the point cloud completion network to obtain the completed point cloud that can better capture the overall structure of the human body, that is, increase the point cloud density to make it closer to the real shape of the human body, and provide a more reliable input data basis for subsequent pose estimation. 2) The spatiotemporal dynamic changes of human posture are learned from both global and local levels through a global-local dual-branch network. This fully utilizes the global information of the completed point cloud and the local information of the original millimeter-wave radar sparse point cloud to provide richer features for the final posture prediction. The global-local dual-branch network includes a global branch and a local branch. The global branch receives the completed point cloud as input and extracts the features of the overall human structure through a convolutional neural network (CNN) to grasp the basic outline of the human posture. The local branch processes the original millimeter-wave radar sparse point cloud, focusing on extracting local motion and reflection details to capture subtle changes in various parts of the human body. The features from both branches are then input into a long short-term memory network (LSTM) to extract dynamic characteristics over time. 3) The different features extracted from the global and local branches are integrated through the feature fusion and prediction mechanism. The mechanism first concatenates the global and local features to form a fusion feature vector containing complete information. Then, the fusion feature vector is processed by the multilayer perceptron (MLP) and finally the three-dimensional coordinate sequence of the key joints of the human body is predicted by regression, thereby completing the estimation of human posture.

2. The human pose estimation method based on millimeter-wave radar point clouds according to claim 1, characterized in that, The point cloud completion network performs the following operations on each frame of the input millimeter-wave radar sparse point cloud: The encoder of the point cloud completion network first processes each frame of millimeter-wave radar sparse point cloud into a multilayer perceptron (MLP) to obtain point features containing local geometric information. The MLP independently extracts high-dimensional features for each point in the sparse point cloud, and its weights are shared among all points, thereby effectively learning the local geometric information of the point cloud and forming an initial feature representation for each point. Subsequently, the point features are input into multiple multi-head self-attention blocks. By calculating the dependencies between point features, the global structural information inside the point cloud is captured. Downsampling is performed by combining the sampling of the farthest point, and finally a global feature vector that condenses the global context information is output. The multi-head self-attention block generates attention weights by calculating the query, key, and value similarity between point features, and aggregates the features of all points in a weighted manner, thereby effectively capturing the long-distance dependencies and global structural information inside the point cloud. The decoder of the point cloud completion network first receives a global feature vector and a set of seed points obtained by performing farthest point sampling on the input sparse point cloud or the initially generated coarse point cloud. Then, it performs multiple upsampling operations. In each upsampling stage, the global feature vector is fused with the features of each seed point or the points generated in the previous stage, and a new set of point coordinates is predicted and generated, thereby gradually increasing the point cloud density and refining the point cloud structure, and finally generating a completed point cloud with the target density.

3. The human pose estimation method based on millimeter-wave radar point clouds according to claim 2, characterized in that, The point cloud completion network generates a denser and more complete point cloud based on a global feature vector and a set of initial seed points. The output completed point cloud contains N completed The seed points are derived from a subset of the input sparse point cloud. During the upsampling process, the global feature vector is combined with the features of each seed point through multiple multilayer perceptrons, and more new 3D point coordinates are gradually generated. This process not only increases the number of point clouds, but also optimizes the position of the newly added points, so that the final output completed point cloud can more accurately reflect the real surface morphology of the human body.

4. The human pose estimation method based on millimeter-wave radar point clouds according to claim 3, characterized in that, The encoder will convert a single frame of millimeter-wave radar sparse point cloud P sparse ∈R B×N×3 As input, R is the set of real numbers, B is the batch size, N is the maximum number of points per frame, and 3 represents the xyz 3D coordinates. First, the encoder uses a Multilayer Perceptron (MLP) to perform preliminary feature extraction on each point, mapping the original 3D coordinates to a 64-dimensional feature space and applying the ReLU activation function. Then, multi-head self-attention blocks are used to process the 64-dimensional features to capture long-range dependencies and global structural information within the point cloud. After one multi-head self-attention block, the number of points is reduced to N / 2 using farthest-point sampling. Next, the feature dimension is expanded from 64 to 128 using the MLP, and multi-head self-attention blocks are applied again to enhance global information interaction, outputting 128-dimensional features. Then, farthest-point sampling is used again to downsample the number of points to N / 4. After each self-attention block, residual connections are added and layer normalization is applied to stabilize the effect. Finally, global pooling is used to aggregate all point features output from the second farthest-point sampling, forming a 128-dimensional global feature vector Z. global .

5. The human pose estimation method based on millimeter-wave radar point clouds according to claim 4, characterized in that, The decoder is based on the global feature vector Z. global To gradually generate a complete point cloud, firstly, a multilayer perceptron (MLP) generates the global feature vector Z. global The vector is mapped to an M×3 dimensional vector and reshaped into a (M,3) low-density point cloud, providing a coarse structural basis, where M represents the number of points in the low-density point cloud. These newly generated points are compared with the original sparse point cloud P. sparse Merge the points, and then use farthest point sampling to select a representative subset from the merged set of points, called the seed point P. seed Next, independent multilayer perceptrons (MLPs) process the seed point and Z-axis respectively. global The point feature matrix F is obtained. seed and global shape feature F shape , will F shape Copy to match F seed The number of rows, and with F seed The fused feature matrix F is formed by concatenating features along the feature dimensions. concat Then, the number of points is gradually increased through a multi-stage upsampling process. At each stage, spatial interpolation is used to generate new points and restore more details. In each upsampling stage, a multilayer perceptron (MLP) is used for feature refinement, and ReLU activation function and dropout layer are used to prevent overfitting. Finally, the decoder uses a multilayer perceptron (MLP) to map the obtained high-dimensional point features back to 3D coordinate space to generate the final completed point cloud P. completed .

6. The human pose estimation method based on millimeter-wave radar point clouds according to claim 5, characterized in that, When training the point cloud completion network, minimize the completed point cloud P. completed With the target point cloud P gt The Chamfe distance loss function L CD To optimize network parameters, the loss function is defined as: In the formula, p represents a point in the completed point cloud, and q represents a point in the target point cloud.

7. The human pose estimation method based on millimeter-wave radar point clouds according to claim 6, characterized in that, The global-local dual-branch network processes different forms of point cloud data streams in parallel through global and local branches, so as to comprehensively utilize the structural information of the completed point cloud and the detailed information of the original sparse point cloud, as follows: The global branch receives the completed point cloud P generated by the point cloud completion network. completed As input; the local branch receives the original millimeter-wave radar sparse point cloud P. sparse As input, each frame of the millimeter-wave radar sparse point cloud contains the three-dimensional coordinates, Doppler velocity, and reflection intensity information of each point. Both branches adopt a cascaded structure. First, a convolutional neural network (CNN) with shared weights independently extracts the spatial features of each frame's point cloud, forming a temporally ordered sequence of frame-level spatial features. Then, these temporally ordered sequence of frame-level spatial features are input into their respective long short-term memory (LSTM) networks to learn and capture the dynamic information of human posture changes over time. Finally, the global branch outputs the final hidden state H, which summarizes the overall spatiotemporal features of the structure. global_final The local branch output summarizes the final hidden state H, which encapsulates the local details and spatiotemporal features of motion. local_final .

8. The human pose estimation method based on millimeter-wave radar point clouds according to claim 7, characterized in that, The completed point cloud P completed And the original millimeter-wave radar sparse point cloud P sparse Before entering their respective branches, frame-level preprocessing is performed: the point cloud is an unordered collection. To adapt to convolution operations, the point cloud in each frame is first sorted in descending order according to its x, y, and z coordinates, and then reshaped into a matrix format in row-major order to make it suitable for CNN processing. After preprocessing, the input tensor shape of the global branch is (T,G,G,3), and the input tensor shape of the local branch is (T,L,L,5), where T is the time step, G is the side length of the global branch input matrix, L is the side length of the local branch input matrix, (T,G,G,3) means that the xyz coordinates of each frame are reshaped into a G×G matrix, and (T,L,L,5) means that the xyz coordinates, Doppler velocity, and reflection intensity information are encoded using an L×L×5 tensor. Each branch employs a cascaded convolutional neural network (CNN) and long short-term memory (LSTM) architecture for spatiotemporal feature extraction. A weight-shared CNN independently processes each frame of point cloud data. The CNN consists of several 3×3 convolutional layers, each followed by batch normalization, ReLU activation, and a 20% dropout layer. Subsequently, a global average pooling layer compresses the obtained frame-level spatial features into a 128-dimensional vector feature, which summarizes the spatial information of each frame. The temporally ordered sequence of frame-level spatial features is input into the respective LSTMs. The LSTMs learn the dynamic information of pose changes over time and understand how the pose evolves. Each LSTM outputs its final 128-dimensional hidden state, which encodes the comprehensive spatiotemporal context features of the input sequence of its corresponding branch.

9. A human pose estimation method based on millimeter-wave radar point clouds according to claim 8, characterized in that, The feature fusion and prediction mechanism will output the final hidden state H from the global branch. global_final The final hidden state H of the local branch output local final By concatenating along the feature dimension, a fused feature vector F containing two complementary pieces of information is generated. fused : F fused =Concat(H global_final ,H local_final ) Subsequently, the feature vector F will be fused. fused The input is fed into a prediction head, which is a multilayer perceptron (MLP) consisting of two fully connected layers. This prediction head directly regresses the total number of N cells in the human body. J A sequence of coordinates of predefined key points in three-dimensional space in This represents the three-dimensional coordinates of the i-th predicted joint.

10. A human pose estimation method based on millimeter-wave radar point clouds according to claim 9, characterized in that, The training process of the pose estimation model aims to minimize the predicted keypoint coordinate sequence J. pred Corresponding ground real key point coordinate sequence Mean square error loss L between MSE , where (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th real joint; the mean square error loss L MSE Defined as:

Images