Millimeter wave radar three-dimensional human body posture estimation method and system
By employing a parallel dual-path feature extraction network and a temporal modeling method, the positioning accuracy and stability issues of millimeter-wave radar 3D human pose estimation methods in complex environments are addressed. This results in high-precision and robust pose estimation, applicable to human-computer interaction, medical rehabilitation, and intelligent security.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-03-10
AI Technical Summary
Existing millimeter-wave radar 3D human pose estimation methods suffer from low positioning accuracy and poor stability. Their performance degrades significantly, especially in complex environments such as low light, occlusion, or smoke. Furthermore, existing methods are either insensitive to local patterns or sensitive to point cloud permutation variance, resulting in poor joint point positioning accuracy and robustness.
A parallel dual-path feature extraction network is adopted, which combines geometric structure and regularized representation to extract features. Then, the network performs temporal modeling through adaptive sliding window segmentation, bidirectional recurrent neural network and attention mechanism to generate global spatiotemporal feature vectors. Finally, the three-dimensional vector of human skeletal joint points is regressed and output.
It achieves high-precision and robust attitude estimation in complex environments, improving positioning accuracy and enhancing system stability, and is applicable to fields such as human-computer interaction, medical rehabilitation, and intelligent security.
Smart Images
Figure CN121634089A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of three-dimensional human posture estimation, in particular to a millimeter wave radar three-dimensional human posture estimation method and system. BACKGROUND
[0002] At present, human posture estimation is a core technology in the field of computer vision and intelligent perception, and is widely used in human-computer interaction, medical rehabilitation, security monitoring and virtual reality fields. The current mainstream scheme relies on optical sensors such as RGB cameras, depth cameras, etc., but such methods have a significant performance decline in complex environments such as dark light, strong light, occlusion or smoke, and there is a risk of privacy leakage. The scheme based on wearable sensors is not affected by light, but it needs the user to wear the device, which is cumbersome to use and has poor comfort, and it is difficult to realize real-time daily monitoring. As a non-contact sensing technology, millimeter wave radar has the advantages of penetrating non-metallic occlusion, all-weather working and protecting privacy, providing a new path for human posture estimation. It can generate point cloud data containing distance, speed, azimuth and elevation information.
[0003] However, millimeter wave radar point cloud has inherent limitations: sparse data, low signal-to-noise ratio due to multipath effect and clutter interference, and uneven point cloud distribution due to differences in human body part reflection, which makes it a serious challenge to stably recover fine three-dimensional posture from millimeter wave radar point cloud. Moreover, the existing solutions have obvious shortcomings: 1) the method based on single-frame point cloud cannot utilize the temporal context; 2) the method of accumulating multiple frames of point cloud is prone to motion blur; 3) the method based on recurrent neural network is insufficient for modeling long-range dependencies. In addition, existing methods mostly use a single feature extractor, such as point set-based networks or regularized representation-based convolutional networks, which are not sensitive to local patterns or sensitive to point cloud permutation variance, resulting in poor positioning accuracy and robustness for joints with small reflection cross sections such as wrists and ankles.
[0004] For the related technologies in the above, the inventors find that the existing millimeter wave radar three-dimensional human posture estimation method has the problems of low positioning accuracy and poor stability. SUMMARY
[0005] In order to improve the positioning accuracy and stability of the millimeter wave radar three-dimensional human posture estimation method, the present application provides a millimeter wave radar three-dimensional human posture estimation method and system.
[0006] In a first aspect, the present application provides a millimeter wave radar three-dimensional human posture estimation method.
[0007] The present application is realized by the following technical solutions:
[0008] A millimeter wave radar three-dimensional human posture estimation method, comprising the following steps,
[0009] Acquire temporal point cloud data of the human body obtained from millimeter-wave radar reflection;
[0010] The time-series point cloud data is expanded into several consecutive segments, each segment containing at least two frames of point cloud data;
[0011] For each frame of point cloud data in each segment, a preset parallel dual-path feature extraction network is used for processing. The first path extracts features based on the geometric structure of the point set, and the second path extracts features based on the regularized representation of the point cloud.
[0012] The features extracted by the two paths of the parallel dual-path feature extraction network are fused to obtain the fused spatial feature vector of each frame of point cloud data.
[0013] In chronological order, the fused spatial feature vectors of all frames in each segment are input into a preset enhanced temporal modeling network. After encoding, global context weighting, and adaptive aggregation, a fixed-length global spatiotemporal feature vector for each segment is output.
[0014] Based on the global spatiotemporal feature vector, a three-dimensional vector representing the joints of the human skeleton is determined.
[0015] In a preferred embodiment, this application can be further configured such that: the parallel dual-path feature extraction network includes a geometric feature extraction network, wherein the geometric feature extraction network adopts a three-layer PointNet++ structure, wherein...
[0016] The first level of sampling uses 32 points, the ball query radius is 0.1m, and the MLP channels are [32,64,128].
[0017] The second-level sampling uses 16 points, with a ball query radius of 0.2m and MLP channels of [64, 128, 256].
[0018] The third level of sampling uses 8 points, the ball query radius is 0.4m, and the MLP channels are [128,256,512].
[0019] In a preferred embodiment, this application can be further configured such that: the parallel dual-path feature extraction network includes a regular feature extraction network, the regular feature extraction network takes an 8×8×4 grid as input, passes through two convolutional blocks in sequence, and is followed by a normalization layer and an activation function after each convolutional block, and outputs a 512-dimensional feature vector.
[0020] In a preferred embodiment, the present application may be further configured such that the enhanced temporal modeling network comprises a bidirectional recurrent neural network layer, a multi-head self-attention layer, and an attention pooling layer connected in series.
[0021] The bidirectional recurrent neural network is a bidirectional gated recurrent unit or a bidirectional long short-term memory network.
[0022] The number of heads in the multi-head self-attention layer is 4-8, and the dimension of each head is 32 to 128;
[0023] The attention pooling layer adaptively aggregates temporal features through a learnable weighting mechanism.
[0024] In a preferred embodiment, this application can be further configured such that the step of determining the three-dimensional vector representing the joint points of the human skeleton based on the global spatiotemporal feature vector includes:
[0025] The global spatiotemporal feature vector is input into a preset regression network, and a three-dimensional vector representing the joints of the human skeleton is output.
[0026] The regression network includes two fully connected layers. The first fully connected layer is followed by a ReLU activation function and a normalization layer. In the first fully connected layer, some neurons are randomly turned off with a probability of 0.3. The second fully connected layer linearly outputs a three-dimensional vector of human skeletal joints.
[0027] In a preferred embodiment, this application can be further configured such that the step of expanding the time-series point cloud data into several consecutive segments includes,
[0028] The time-series point cloud data includes three-dimensional spatial coordinates and Doppler velocity information. Based on the three-dimensional spatial coordinates and Doppler velocity information, the average point cloud density or average Doppler velocity of the segment to be expanded is determined.
[0029] An adaptive sliding window mechanism is adopted to dynamically adjust the window length according to the average density of the point cloud or the average Doppler velocity, thereby expanding the time-series point cloud data into several continuous segments.
[0030] In a preferred embodiment, this application can be further configured such that the step of dynamically adjusting the window length based on the average point cloud density or the average Doppler velocity includes:
[0031] When the average point cloud density is below a threshold or the average Doppler velocity increases, the window length is increased.
[0032] When the average density of the point cloud is higher than a threshold or the average Doppler velocity decreases, the window length is reduced.
[0033] In a preferred example, this application can be further configured as follows: the first path extracts features based on the geometric structure of the point set, and the second path extracts features based on the regularized representation of the point cloud.
[0034] The first path uses a neural network based on graph convolution or point set learning to explicitly model the geometric relationships of points in the point cloud and extract features;
[0035] The second path sorts the disordered point cloud according to its spatial coordinates and reshapes it into a two-dimensional matrix, then extracts its regularized features through a convolutional neural network.
[0036] In a preferred embodiment, this application can be further configured as follows: the step of fusing the features extracted by the two paths of the parallel dual-path feature extraction network to obtain the fused spatial feature vector of each frame of point cloud data includes...
[0037] When the feature dimensions extracted by the two paths are different, the feature vector dimensions output by the two paths are made consistent through a fully connected layer, and then a weighted summation method is used to fuse the features to obtain the fused spatial feature vector of each frame of point cloud data.
[0038] When the feature dimensions extracted by the two paths are the same, the features extracted by the first path and the features extracted by the second path are concatenated to obtain the concatenated result. Then, the fused spatial feature vector of each frame of point cloud data is obtained by projection through a fully connected layer.
[0039] Secondly, this application provides a millimeter-wave radar three-dimensional human posture estimation system.
[0040] This application is achieved through the following technical solution:
[0041] A millimeter-wave radar three-dimensional human pose estimation system includes,
[0042] The millimeter-wave radar acquisition module is used to transmit FMCW millimeter-wave signals and receive the echo signals reflected from the target.
[0043] The signal processing and point cloud generation module is used to perform three-dimensional FFT processing and CFAR detection on the echo signal to generate time-series point cloud data.
[0044] The inference module includes a memory and a processor, the memory storing a computer program that, when executed by the processor, is used to implement the steps of any of the methods described above.
[0045] The results output and application interface module is used to output and transmit the obtained three-dimensional vectors of human skeletal joints.
[0046] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:
[0047] A parallel heterogeneous dual-path feature extraction architecture is introduced, which can deeply mine the spatial information contained in radar point clouds from two complementary perspectives: unstructured point sets and regularized two-dimensional representations. The geometric structure path focuses on capturing the topological relationships between human joints, while the dense feature path enhances the sensitivity to subtle local postures such as hands and feet. Feature fusion of the two paths effectively overcomes the inherent limitations of a single feature extractor, achieving a more comprehensive and robust feature representation of human postures, resulting in more complete feature extraction. Furthermore, thanks to the synergistic effect of multi-branch spatial features and enhanced temporal modeling networks… This system can fully utilize the geometric structure information of point clouds and the temporal dynamic information during motion, greatly improving estimation accuracy. At the same time, the multi-head self-attention mechanism can effectively distinguish between valid signals and noise, ensuring stable performance even in complex environments such as severe occlusion or multipath interference. The combination of bidirectional recurrent neural networks and attention mechanisms enables the model to have excellent adaptability and generalization ability for motions of different speeds and styles. In addition, multi-branch feature learning enables the model to understand human posture from different perspectives, thus showing strong generalization performance for user body shapes and action types that have never been seen before, and comprehensively enhancing the system's stability and robustness. Attached Figure Description
[0048] Figure 1 This is a schematic diagram of the main process of a millimeter-wave radar three-dimensional human pose estimation method provided as an exemplary embodiment of this application.
[0049] Figure 2 This is a visualization of a millimeter-wave radar three-dimensional human pose estimation method provided as another exemplary embodiment of this application. Detailed Implementation
[0050] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.
[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0052] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.
[0053] To address the issues of low positioning accuracy and poor stability in existing millimeter-wave radar 3D human posture estimation methods due to poor environmental adaptability of optical schemes, sparse point clouds, noise interference, and difficulties in complex motion modeling, this application employs adaptive sliding window segmentation of the radar temporal point cloud; extracts and fuses spatial features from the point set geometry and regularized representations using a parallel dual-path network; utilizes a bidirectional recurrent neural network combined with an attention mechanism for temporal modeling to generate feature vectors that fuse global spatiotemporal information; and finally regresses and outputs the 3D coordinates of human joints. This approach enables high-precision and robust posture estimation in low-light and occluded environments, while also offering privacy protection advantages, making it suitable for human-computer interaction, medical rehabilitation, and intelligent security.
[0054] This application provides a method for estimating three-dimensional human posture using millimeter-wave radar. The main steps of the method are described below.
[0055] Acquire temporal point cloud data of the human body obtained from millimeter-wave radar reflection;
[0056] The time-series point cloud data is expanded into several consecutive segments, each segment containing at least two frames of point cloud data;
[0057] For each frame of point cloud data in each segment, a preset parallel dual-path feature extraction network is used for processing. The first path extracts features based on the geometric structure of the point set, and the second path extracts features based on the regularized representation of the point cloud.
[0058] The features extracted by the two paths of the parallel dual-path feature extraction network are fused to obtain the fused spatial feature vector of each frame of point cloud data.
[0059] In chronological order, the fused spatial feature vectors of all frames in each segment are input into a preset enhanced temporal modeling network. After encoding, global context weighting, and adaptive aggregation, a fixed-length global spatiotemporal feature vector for each segment is output.
[0060] Based on the global spatiotemporal feature vector, a three-dimensional vector representing the joints of the human skeleton is determined.
[0061] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.
[0062] Reference Figure 1To acquire temporal point cloud data of the human body obtained from millimeter-wave radar reflection.
[0063] For each frame of temporal point cloud data, data preprocessing is performed. First, the points are sorted lexicographically according to the x→y→z priority. Then, the 64 points are reshaped into an 8×8 two-dimensional grid to form a regularized representation, while the original point cloud is retained for geometric feature extraction.
[0064] Next, the temporal point cloud data is expanded into several consecutive segments, each segment containing at least two frames of point cloud data.
[0065] In one embodiment, the time-series point cloud data includes three-dimensional spatial coordinates and Doppler velocity information.
[0066] Temporal point cloud data can also include point cloud reflection intensity information.
[0067] Based on the three-dimensional spatial coordinates and Doppler velocity information, determine the average point cloud density or average Doppler velocity of the segment to be expanded.
[0068] An adaptive sliding window mechanism is adopted to dynamically adjust the window length according to the average density of the point cloud or the average Doppler velocity, thereby expanding the time-series point cloud data into several continuous segments.
[0069] Each expanded segment contains L frames of point cloud data, where L is a positive integer greater than 1.
[0070] In this embodiment, the initial length L of the adaptive sliding window is a positive integer in the range of 8-16. Preferably, L = 12.
[0071] In one embodiment, the step of dynamically adjusting the window length based on the average point cloud density or the average Doppler velocity includes,
[0072] When the average point cloud density is below a threshold or the average Doppler velocity increases, the window length is increased.
[0073] When the average density of the point cloud is higher than a threshold or the average Doppler velocity decreases, the window length is reduced.
[0074] An adaptive mechanism dynamically adjusts the window length based on the average point cloud density or average Doppler velocity of the fragment. When the point cloud density is below a threshold or the average velocity increases, L is increased to introduce more temporal context; conversely, L is decreased to reduce computational latency.
[0075] Furthermore, for each frame of point cloud data in each segment, a preset parallel dual-path feature extraction network is used for processing, wherein the first path extracts features based on the geometric structure of the point set, and the second path extracts features based on the regularized representation of the point cloud.
[0076] In one embodiment, the parallel dual-path feature extraction network includes a geometric feature extraction network, which adopts a three-layer PointNet++ structure and outputs 512-dimensional global features.
[0077] The first level of sampling uses 32 points, the ball query radius is 0.1m, and the MLP channels are [32,64,128].
[0078] The second-level sampling uses 16 points, with a ball query radius of 0.2m and MLP channels of [64, 128, 256].
[0079] The third level of sampling uses 8 points, the ball query radius is 0.4m, and the MLP channels are [128,256,512].
[0080] In one embodiment, the parallel dual-path feature extraction network includes a regular feature extraction network. The regular feature extraction network takes an 8×8×4 grid as input and passes it through two convolutional blocks sequentially. Each convolutional block is followed by a normalization layer (BatchNorm) and a ReLU activation function, and the output is a 512-dimensional feature vector. The convolutional kernel of each convolutional block is 3×3, and the number of channels ranges from 16 to 32.
[0081] The geometric feature extraction network is based on a neural network that learns from graph convolution or point sets. It explicitly models the geometric relationships between points in a point cloud and extracts features.
[0082] The regular feature extraction network sorts the disordered point cloud according to its spatial coordinates and reshapes it into a two-dimensional matrix, and then extracts its regularized features through a convolutional neural network.
[0083] Specifically, in this embodiment, ten representative movements were recorded from eight subjects of varying body types, including upper limb extension, trunk twisting, squatting, lunging, and lateral movement. These movements were repeated at different speeds to simulate real-world movement variations and construct a dataset. The eight subjects were four men and four women, with heights ranging from 155 to 190 cm and BMIs from 18 to 28 kg / m². 2 A total of over 300,000 frames of synchronous temporal point cloud data were collected.
[0084] The dataset was divided into a training set (70%), a validation set (15%), and a test set (15%) based on temporal coherence. The geometric feature extraction network and the regular feature extraction network were trained using the constructed training set, and the model weights were adjusted to minimize the loss function.
[0085] After each training session, the model performance is evaluated using a validation set, and hyperparameters such as the learning rate and regularization coefficient are adjusted to prevent overfitting.
[0086] After training is complete, the model's estimation accuracy is evaluated using a test set to determine if it meets the requirements.
[0087] In addition, occlusion test sets and cross-subject generalization test sets were specifically built to systematically evaluate the robustness of the model in actual deployment.
[0088] Furthermore, the features extracted from the two paths of the parallel dual-path feature extraction network are fused to obtain the fused spatial feature vector of each frame of point cloud data.
[0089] When the feature dimensions extracted by the two paths are different, the feature vector dimensions output by the two paths are made consistent through a fully connected layer, and then a weighted summation method is used to fuse the features to obtain the fused spatial feature vector of each frame of point cloud data.
[0090] When the feature dimensions extracted by the two paths are the same, the features extracted by the first path and the features extracted by the second path are concatenated to obtain the concatenated result. Then, the fused spatial feature vector of each frame of point cloud data is obtained by projection through a fully connected layer.
[0091] The fused spatial feature vector is the spatial feature of each frame. It contains rich spatial structure information within a single frame, but does not contain temporal motion information across frames.
[0092] Next, in chronological order, the fused spatial feature vectors of all frames in each segment are input into a preset enhanced temporal modeling network. After encoding, global context weighting, and adaptive aggregation operations, a fixed-length global spatiotemporal feature vector for each segment is output.
[0093] Specifically, the enhanced temporal modeling network includes a bidirectional recurrent neural network layer, a multi-head self-attention layer, and an attention pooling layer connected in series.
[0094] The bidirectional recurrent neural network is a bidirectional gated recurrent unit or a bidirectional long short-term memory network.
[0095] The number of heads in the multi-head self-attention layer is 4-8, and the dimension of each head is 32 to 128;
[0096] The attention pooling layer adaptively aggregates temporal features through a learnable weighting mechanism.
[0097] The input parameters of the augmented temporal modeling network are the fused spatial feature vector sequence of all frames in each segment, with each vector having a dimension of 512 and an input data shape of [batch size, 12, 512]. The training labels are the ground truth key point 3D coordinates of each segment, i.e., a 75-dimensional vector corresponding to the 3D coordinates of 25 key points. The training labels are consistent with the final output target of the entire augmented temporal modeling network. The output parameter is a 256-dimensional real number vector, i.e., the "compact feature vector". This vector, as a condensed representation of the spatiotemporal information of the entire segment, is fed into the subsequent regression network.
[0098] Training labels are the ground truth labels required for model training and evaluation. The system synchronously integrates a Microsoft Kinect V2 sensor, using the output 25 keypoint 3D coordinates as supervisory signals. During deployment, the radar is installed at a height of 2 meters and tilted downwards at 30° to optimize coverage; the Kinect is installed at a height of 1 meter and kept horizontally aligned with the radar. Hardware synchronization signals and software timestamps are aligned to ensure temporal consistency among multi-source data.
[0099] When training the augmented temporal modeling network, the AdamW optimizer was used with an initial learning rate of 0.001, a batch size of 64, and a cosine annealing scheduling strategy, and the training lasted for 200 epochs.
[0100] To prevent overfitting, Dropout was applied after the BiGRU layer with an inactivation rate of 0.3. By randomly shutting down some neurons during training, the model learns on different simplified versions of the model each time it is trained, resulting in a more stable learning outcome for the final model.
[0101] The enhanced temporal modeling network adopts an end-to-end joint optimization training paradigm, which differs from the traditional method of training feature extraction, temporal modeling and regression in stages. By training the temporal module in a unified manner with other parts of the network, the temporal model can directly learn the accurate mapping from the original point cloud to spatiotemporal features. Its embedded self-attention mechanism can implicitly learn the ability to focus on key frames and key features without manual intervention. Combining the local fine modeling of BiGRU and the global context capture of self-attention, it effectively overcomes the shortcomings of traditional recurrent networks in long sequence modeling.
[0102] Enhanced temporal modeling networks encode temporal sequences and weight them with global context to output a fixed-length, compact feature vector that represents the spatiotemporal dynamics of the entire segment, namely the global spatiotemporal feature vector.
[0103] The compact feature vector is a global representation obtained by adaptively aggregating features from all time steps through an attention pooling layer. Its dimension is much smaller than the sum of the feature dimensions of the original input sequence, achieving information condensation and facilitating subsequent regression tasks. By enhancing the temporal modeling network, the fused spatial feature vectors of all frames within a segment are integrated to capture motion trajectories and state transitions, ultimately generating a "compact feature vector" that can represent the global spatiotemporal context of the entire segment. This represents a sublimation from "frame-level spatial features" to "segment-level spatiotemporal features," a key improvement for enhancing the accuracy of continuous action estimation.
[0104] Finally, based on the global spatiotemporal feature vector, a three-dimensional vector representing the joints of the human skeleton is determined, including:
[0105] The global spatiotemporal feature vector is input into a preset regression network, and a three-dimensional vector representing the joints of the human skeleton is output.
[0106] The regression network includes two fully connected layers. The first fully connected layer (256→128) is followed by a ReLU activation function and a normalization layer. In the first fully connected layer, some neurons are randomly turned off with a probability of 0.3. The second fully connected layer (128→75) linearly outputs the three-dimensional vector of the human skeletal joints.
[0107] The training method for the regression network is the same as the existing method, except that the loss function of the regression network is set to a weighted mean square error function, which can assign different weight coefficients to joints of different categories in order to balance the learning objectives.
[0108] The compact feature vector is input into the regression network, which outputs a 3K-dimensional vector representing the three-dimensional coordinates of K skeletal joints in the human body. The number of skeletal joints is K = 25.
[0109] Reference Figure 2 This paper demonstrates the visualization effect of a millimeter-wave radar 3D human pose estimation method in typical scenarios. It can accurately reconstruct the human skeleton, including using ground truth labels as a comparison benchmark.
[0110] The accuracy indicators of the radar point cloud are as follows: the average positioning error of 25 joints is 1.82cm, of which the error of the core trunk joints is less than 1.5cm, and the error of the fine joints such as the wrist and ankle joints is controlled within 2.5cm.
[0111] Model robustness: In a simulated occlusion scenario with 30% point cloud loss, the performance degradation is less than 15%; the performance fluctuation range is less than 10% at different motion speeds;
[0112] Efficiency assessment: On the embedded platform (NVIDIA Jetson Xavier NX), the single-frame inference time is <20ms, meeting the requirements of real-time applications (≥20FPS), and the average system power consumption is <10W, demonstrating potential for mobile deployment.
[0113] Experimental results show that, under various test scenarios, a millimeter-wave radar 3D human pose estimation method can maintain an average positioning error of 1.7 to 2.0 cm for 25 joint points. Compared with the traditional single-architecture method, the positioning accuracy is improved by about 15% to 30%.
[0114] In summary, a millimeter-wave radar 3D human pose estimation method introduces a parallel heterogeneous dual-path feature extraction architecture, which can deeply mine the spatial information contained in radar point clouds from two complementary perspectives: unstructured point sets and regularized 2D representations. The geometric structure path focuses on capturing the topological relationships between human joints, while the dense feature path enhances the sensitivity to subtle local poses such as hands and feet. The feature fusion of the two paths effectively overcomes the inherent limitations of a single feature extractor, achieving a more comprehensive and robust feature representation of human pose, resulting in more complete feature extraction. Furthermore, thanks to the multi-branch spatial feature and enhanced temporal modeling network... Through collaborative network operation, a millimeter-wave radar 3D human pose estimation method can fully utilize the geometric structure information of point clouds and the temporal dynamic information during the motion process, greatly improving the estimation accuracy. At the same time, the multi-head self-attention mechanism can effectively distinguish between effective signals and noise, ensuring stable performance even in complex environments such as severe occlusion or multipath interference. The combination of bidirectional recurrent neural networks and attention mechanisms enables the model to have excellent adaptability and generalization ability for motions of different speeds and styles. In addition, multi-branch feature learning enables the model to understand human pose from different perspectives, thus showing strong generalization performance for user body shapes and action types that have not been seen before, and comprehensively enhancing the robustness of the system.
[0115] A millimeter-wave radar 3D human pose estimation method achieves high-precision and robust human skeleton estimation by parallel dual-path feature extraction, fusing heterogeneous features with enhanced spatiotemporal modeling, bidirectional temporal modeling and attention mechanism.
[0116] A millimeter-wave radar-based three-dimensional human pose estimation method eliminates the risk of privacy leakage by eliminating the need to collect any visual biometric features such as facial features or body shape. At the same time, the entire system is optimized with an average operating power consumption of less than 600mW, making it extremely low-power and very suitable for long-term, uninterrupted deployment in practical scenarios with high privacy sensitivity, such as smart homes and digital healthcare. It combines privacy protection with practical value.
[0117] A parallel dual-path fusion framework for a millimeter-wave radar 3D human pose estimation method adopts a modular design, which has good flexibility and can easily integrate data from other sensors such as inertial measurement units and Wi-Fi channel state information, laying a solid technical foundation for building a more powerful and robust multimodal perception system in the future.
[0118] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0119] This application also provides a millimeter-wave radar three-dimensional human pose estimation system, which can be applied to any of the millimeter-wave radar three-dimensional human pose estimation methods described in the above embodiments. The millimeter-wave radar three-dimensional human pose estimation system includes...
[0120] The millimeter-wave radar acquisition module is used to transmit FMCW millimeter-wave signals and receive the echo signals reflected from the target.
[0121] The signal processing and point cloud generation module is used to perform three-dimensional FFT processing and CFAR detection on the echo signal to generate time-series point cloud data.
[0122] The inference module includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the steps of any of the above methods.
[0123] The results output and application interface module is used to output and transmit the obtained three-dimensional vectors of human skeletal joints.
[0124] The millimeter-wave radar acquisition module employs a highly integrated radio frequency architecture. Its core includes a RF front-end chip with four transmit and four receive channels and a dedicated signal processor. The front-end chip operates in the 60GHz band. Key radar parameters, after optimization, are set as follows: starting frequency 59.5GHz, bandwidth 3.22GHz, chirp duration 80μs, 64 chirs per frame, and frame period 50ms. Based on this configuration, the system theoretically achieves a range resolution of 5.00cm, a velocity resolution of 0.13m / s, and an azimuth resolution of 14.32°. The radar communicates with the host computer via a UART-to-USB interface, with average power consumption controlled below 570mW, meeting long-term deployment requirements. The position and data flow of the millimeter-wave radar acquisition module within the entire system are as follows: Figure 1 As shown.
[0125] After system startup, the millimeter-wave radar continuously acquires data and generates a point cloud stream through the signal processing and point cloud generation modules. The inference module adopts a pipelined operation: while performing inference in the current window, it preloads data for the next window. Finally, it outputs the 3D coordinates, confidence scores, and timestamps of the joints. The results output are the 3D vectors of the human skeletal joints obtained from the output and transmission of the application interface module.
[0126] Upper-layer applications of the system, such as virtual reality, rehabilitation assessment, and intelligent security, can obtain skeleton data in real time through the provided API interface, achieving seamless integration.
[0127] A millimeter-wave radar 3D human pose estimation system provides a complete solution for 3D human pose estimation that is high-precision, robust, low-power, and privacy-preserving, laying a reliable technical foundation for next-generation ubiquitous computing and intelligent sensing applications.
[0128] For specific limitations regarding a millimeter-wave radar 3D human pose estimation system, please refer to the limitations of a millimeter-wave radar 3D human pose estimation method mentioned above, which will not be repeated here.
[0129] The modules in the aforementioned millimeter-wave radar three-dimensional human pose estimation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.
[0130] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements any of the millimeter-wave radar three-dimensional human pose estimation methods described above.
[0131] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the millimeter-wave radar three-dimensional human pose estimation methods described above.
[0132] In one embodiment, a computer program product is provided, which includes a computer program that, when executed by a processor, implements any of the millimeter-wave radar three-dimensional human pose estimation methods described above.
[0133] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. When executed, the computer program may include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0134] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.
Claims
1. A method for 3D human pose estimation from millimeter wave radar, characterized in that, The method comprises the following steps, acquiring time-series point cloud data of a human body reflected by a millimeter wave radar; extending the time-series point cloud data into a plurality of continuous segments, each of the segments containing at least two frames of point cloud data; for each frame of point cloud data in each segment, processing the frame of point cloud data by using a preset parallel double-path feature extraction network, wherein a first path extracts features based on the geometric structure of a point set, and a second path extracts features based on the regularized representation of a point cloud; performing fusion operation on the features extracted by the two paths of the parallel double-path feature extraction network to obtain a fusion spatial feature vector of each frame of point cloud data; inputting the fusion spatial feature vectors of all frames in each segment in time sequence into a preset enhanced time-series modeling network, and outputting a fixed-length global spatio-temporal feature vector of each segment through encoding, global context weighting and adaptive aggregation operation; determining a three-dimensional vector representing a human body skeletal joint based on the global spatio-temporal feature vector. 2.The method of claim 1, wherein, The parallel double-path feature extraction network comprises a geometric feature extraction network, and the geometric feature extraction network adopts a three-level PointNet++ structure, wherein the first level samples 32 points, the spherical query radius is 0.1 m, and the MLP channel is [32, 64, 128]; the second level samples 16 points, the spherical query radius is 0.2 m, and the MLP channel is [64, 128, 256]; the third level samples 8 points, the spherical query radius is 0.4 m, and the MLP channel is [128, 256, 512]. 3.The method of claim 1, wherein, The parallel double-path feature extraction network comprises a regular feature extraction network, and the regular feature extraction network inputs a 8*8*4 grid, sequentially passes through two convolutional blocks, and outputs a 512-dimensional feature vector after each convolutional block is connected with a normalization layer and an activation function. 4.The method of claim 1, wherein, The enhanced time-series modeling network comprises a bidirectional recurrent neural network layer, a multi-head self-attention layer and an attention pooling layer connected in sequence; the bidirectional recurrent neural network is a bidirectional gated recurrent unit or a bidirectional long short-term memory network; the number of heads of the multi-head self-attention layer is 4-8, and the dimension of each head is 32-128; the attention pooling layer adaptively aggregates time-series features through a learnable weight mechanism.
5. The method of claim 1-4, wherein, The step of determining a three-dimensional vector representing a human body skeletal joint based on the global spatio-temporal feature vector comprises inputting the global spatio-temporal feature vector into a preset regression network to output a three-dimensional vector representing a human body skeletal joint; wherein the regression network comprises two fully connected layers, a ReLU activation function and a normalization layer are connected after the first fully connected layer, a part of neurons in the first fully connected layer is randomly turned off with a probability of 0.3, and the second fully connected layer linearly outputs a three-dimensional vector of a human body skeletal joint. 6.The method of claim 5, wherein, The step of extending the time-series point cloud data into a plurality of continuous segments comprises the time-series point cloud data comprises three-dimensional spatial coordinates and Doppler velocity information, and the average density or average Doppler velocity of the point cloud of the segment to be extended is determined according to the three-dimensional spatial coordinates and Doppler velocity information. The adaptive sliding window mechanism is adopted to dynamically adjust the window length according to the point cloud average density or the average Doppler velocity, and the time-series point cloud data is expanded into several continuous segments.
7. The method of claim 6, wherein, The step of dynamically adjusting the window length according to the point cloud average density or the average Doppler velocity comprises, When the point cloud average density is lower than a threshold or the average Doppler velocity is increased, the window length is increased; When the point cloud average density is higher than a threshold or the average Doppler velocity is decreased, the window length is decreased. 8.The method of claim 5, wherein, The steps of extracting features based on the geometric structure of the point set in the first path and extracting features based on the regularized representation of the point cloud in the second path comprise, The first path adopts a neural network based on graph convolution or point set learning to explicitly model the geometric relationship of points in the point cloud and extract features; The second path sorts the unordered point cloud according to the spatial coordinates and reshapes it into a two-dimensional matrix, and extracts its regularized features through a convolutional neural network. 9.The method of claim 5, wherein, The steps of performing fusion operation on the features extracted by the two paths of the parallel double-path feature extraction network to obtain the fusion spatial feature vector of each frame of point cloud data comprise, When the feature dimensions extracted by the two paths are different, the feature vectors output by the two paths are made to have consistent dimensions through a fully connected layer, and then a weighted summation method is used for feature fusion to obtain the fusion spatial feature vector of each frame of point cloud data; When the feature dimensions extracted by the two paths are the same, the features extracted by the first path and the features extracted by the second path are spliced to obtain a splicing result, and then a fully connected layer is used for projection to obtain the fusion spatial feature vector of each frame of point cloud data.
10. A millimeter wave radar three-dimensional human pose estimation system, characterized by, The method comprises, The millimeter wave radar acquisition module is configured to emit FMCW millimeter wave signals and receive echo signals reflected by targets; The signal processing and point cloud generation module is configured to perform three-dimensional FFT processing and CFAR detection on the echo signals to generate time-series point cloud data; The inference module comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 9; The result output and application interface module is configured to output and transmit the three-dimensional vector of the human skeleton joint point obtained.