A method for human pose estimation from millimeter-wave radar point cloud based on velocity correction

Abstract

The present invention relates to the field of computer vision technology, and more specifically to a method for human pose estimation from millimeter-wave radar point clouds based on velocity correction, comprising: obtaining radar point cloud data to be estimated, performing multi-frame point cloud aggregation on the radar point cloud data in the form of a sliding window, thereby generating radar point cloud input data; inputting the radar point cloud input data into a trained rough estimation network branch model and a spatial velocity feature fine-tuning network model; and fine-tuning the rough human skeletal point positions output by the rough estimation network branch model based on the spatial velocity features output by the spatial velocity feature fine-tuning network model, thereby generating accurate human skeletal point positions. The present invention fully utilizes radar point cloud information and inter-frame compensatory features, while further improving model performance through fine correction of spatial and velocity information, thereby achieving high-performance radar signal human pose estimation.

Description

Technical Field

[0001] The present invention relates to the technical field of human posture prediction, and in particular to a millimeter-wave radar point cloud human posture estimation method based on velocity correction. Background Art

[0002] While image and video-based 3D human pose estimation has made significant progress in numerous scenarios, it faces a fundamental challenge: images and videos themselves only provide two-dimensional information about the human body. This leads to uncertainty in predicting three-dimensional information, especially in complex scenarios such as occlusion, varying illumination, and image perspective. In such cases, relying solely on image and video data may not fully restore the true pose of the human body in three-dimensional space, limiting the accuracy and robustness of its application in real-world scenarios.

[0003] At night, due to limited lighting conditions, image information obtained by image acquisition devices can be underexposed, overexposed, and blurred by motion, making human pose estimation more difficult and less effective. Furthermore, in some privacy scenarios, image-based human pose estimation cannot guarantee privacy.

[0004] Millimeter-wave radar, as a non-line-of-sight sensing technology, has the advantages of being able to penetrate optically opaque objects and being less affected by lighting conditions and occlusions. In recent years, human pose estimation based on radar signals has become a new research direction. The basic principle is to accurately estimate key points and pose information of the human skeleton by analyzing the human body signal reflected by radar. This method also has the advantage of being robust in diverse environments. However, due to the weak nature of radar signals and their susceptibility to interference, the accuracy of its pose estimation has been difficult to achieve and can replace image-based human pose estimation. Summary of the Invention

[0005] In view of the shortcomings of existing technologies, this paper provides a method for human pose estimation from millimeter-wave radar point clouds based on velocity correction. This method primarily utilizes an encoder-decoder structure to obtain a rough estimate of human joint positions and further corrects these positions using velocity characteristics. This results in a high-performance radar-based human pose estimation model that accurately predicts human pose from radar signals.

[0006] The technical means adopted in the present invention are as follows:

[0007] A method for estimating human body posture from millimeter-wave radar point cloud based on velocity correction, characterized by comprising the following steps:

[0008] Obtain the radar point cloud data to be estimated, perform multi-frame point cloud aggregation on the radar point cloud data in the form of a sliding window, and thus generate radar point cloud input data;

[0009] The radar point cloud input data is respectively input into the trained coarse estimation network branch model and the spatial velocity feature fine-tuning network model; the coarse estimation network branch model is constructed based on the human sequence codec structure and is used to roughly estimate the position of human skeletal points. The coarse estimation network branch model includes a point cloud voxelization module, two GRU encoding units, an attention layer and a GRU decoder; the spatial velocity feature fine-tuning network model is constructed based on 3D convolution and long short-term memory network, and is used to extract features from the velocity data projected in the point cloud voxelization space to obtain spatial velocity features;

[0010] The rough estimated human skeleton point positions output by the rough estimation network branch model are fine-tuned based on the spatial velocity features output by the spatial velocity feature fine-tuning network model, thereby generating accurate human skeleton point positions.

[0011] Furthermore, a rough estimate of the position of human skeleton points is performed based on the rough estimation network branch model, including:

[0012] Perform voxel mapping encoding on the radar point cloud input data to obtain the sequence corresponding to the point cloud position;

[0013] Learning the encoding representation of the sequence through the GRU encoding unit;

[0014] The attention layer allows the learning process to focus on the final encoder state while also focusing on the intermediate encoder state, thereby improving the efficiency of the decoder.

[0015] The decoder predicts the final human skeleton point sequence encoding representation;

[0016] The human skeleton encoding sequence output by the decoder is devoxelized to obtain a rough estimate of the human skeleton point position.

[0017] Furthermore, based on the spatial velocity feature, the network model is fine-tuned to extract features from the radar point cloud data after splicing with the velocity feature, including:

[0018] Acquire velocity data based on radar point cloud input data, and project the velocity data into point cloud voxelized space to acquire point cloud velocity data;

[0019] Perform feature extraction on the point cloud velocity data through 3D convolution to obtain spatial velocity features;

[0020] The spatial velocity features are input into the LSTM layer to capture the dependencies between consecutive frames of the time series velocity data.

[0021] Furthermore, fine-tuning the rough estimated human skeleton point position output by the rough estimation network branch model based on the spatial velocity feature output by the spatial velocity feature fine-tuning network model includes:

[0022] Use a fully connected layer to align the spatial velocity features with their channels;

[0023] A fully connected layer is used to fine-tune the rough estimate of the human skeleton point position output by the rough estimation network.

[0024] Furthermore, the radar point cloud data is aggregated into multiple frames in the form of a sliding window to generate radar point cloud input data, including:

[0025] In the form of a sliding window, the previous two frames of radar point cloud data and the current frame data are retained each time to fully utilize the continuity features between frames.

[0026] Furthermore, the training steps of the rough estimation network branch model and the spatial velocity feature fine-tuning network model include:

[0027] Acquire human body posture data collected by millimeter wave radar, perform Fourier transform, MTI filtering and CFAR detection on the human body posture data, thereby obtaining human body radar point cloud data;

[0028] Acquire human joint point position data corresponding to the human body radar point cloud data through a Kinect camera;

[0029] The human body radar point cloud data is used as input data and the human body joint point position data is used as output data to train the rough estimation network branch model and the spatial velocity feature fine-tuning network model.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] The overall framework of this invention adopts the concept of rough estimation followed by fine-tuning. The rough estimation network maps the point cloud into a sequence by voxelizing it, fully leveraging the strong learning capabilities of the human sequence codec's serialization relationships. The rough estimation network then obtains a rough estimate of the human joint positions. The fine-tuning network utilizes the radial velocity information obtained by Fourier transforming radar data, effectively extracting the corresponding position and velocity features through 3D convolution, and further fine-tuning and optimizing the network's prediction results. Furthermore, multi-frame aggregated input effectively utilizes the continuity relationship between frames, making the human pose predicted by the model more coherent, thereby improving the overall performance of the millimeter-wave radar-based human pose estimation algorithm.

[0032] Based on the above reasons, the present invention can be widely promoted in fields such as radar human posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 This is a flow chart of a method for estimating human posture from millimeter-wave radar point clouds based on velocity correction in an embodiment of the present invention.

[0035] Figure 2 This is a model architecture diagram of a method for estimating human posture from a millimeter-wave radar point cloud based on velocity correction in an embodiment of the present invention.

[0036] Figure 3 4 is a flowchart of radar signal processing in an embodiment of the present invention.

[0037] Figure 4 This is a flowchart of point cloud voxelization in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0039] like Figure 1 As shown, the present invention provides a method for estimating human body posture from millimeter-wave radar point cloud based on velocity correction, comprising:

[0040] S1. Obtain radar point cloud data to be estimated, perform multi-frame point cloud aggregation on the radar point cloud data in the form of a sliding window, and thus generate radar point cloud input data.

[0041] In this application, the TI IWR6843AOP millimeter-wave radar is used to collect human body posture radar data. The directly collected human body posture radar data is subjected to Fourier transform, MTI filtering, and CFAR detection to obtain the three-dimensional coordinates of the human body radar point data.

[0042] Before using the method of the present invention for posture estimation, the steps of training data acquisition and model training are also included. Specifically, in this application, the TI IWR6843AOP millimeter-wave radar is used to collect human posture radar data, and a Kinect camera is placed at the same height to obtain the corresponding real human joint coordinates. The two are aligned based on the time frame, and the coordinates obtained by the camera are used as supervision data for training the network.

[0043] S2. Input the radar point cloud input data into the trained rough estimation network branch model and the spatial velocity feature fine-tuning network model respectively. The architecture of the rough estimation network branch model and the spatial velocity feature fine-tuning network model is as follows: Figure 2 shown.

[0044] The coarse estimation network branch model is built based on the human sequence encoder-decoder structure. The coarse estimation network branch model includes a point cloud voxelization module, two GRU encoding units, an attention layer, and a GRU decoder. The coarse estimation network branch based on voxelized data and encoder-decoder roughly estimates the position of human skeleton points, including:

[0045] a. Perform voxel mapping encoding on the point cloud data to obtain the sequence corresponding to the point cloud position. Figure 4 The figure below shows the flow chart for voxelization of point clouds. First, the center of the radar point cloud is calculated, and a large cube (length x width x height, 1.7m x 1.4m x 2.2m) is constructed based on the center point to completely encompass the human body point cloud. This large cube is then divided into multiple small cubes with sides of 5cm. These cubes are sorted to ensure that each small cube has a unique serial number. The point cloud is then embedded into the corresponding serial number block based on its actual position. The resulting serial number sequence is the encoding sequence obtained after voxelization. This method not only preserves the spatial information of the point cloud, but also reduces the spatial information of the point cloud from a three-dimensional representation to a one-dimensional sequence.

[0046] b. Learn the encoding representation of the sequence through the GRU encoding unit.

[0047] c. The attention layer improves the efficiency of the decoder by allowing it to focus on the intermediate encoder states while paying attention to the final encoder state.

[0048] d. The decoder predicts the final human skeleton point sequence encoding representation.

[0049] e. Perform devoxel mapping on the human skeleton encoding sequence output by the decoder to obtain a rough estimate of the human skeleton point position.

[0050] Specifically, a coarse estimation network model based on the human sequence encoder-decoder, such as Figure 2 As shown, the coarse estimation network consists of a voxelization module, an embedding layer, an encoder layer, a decoder layer, and an output layer.

[0051] First, the input radar point cloud data has the shape x∈R N×90×3 , where N represents the number of time frames, 90 is the number of points per frame, and 3 represents the spatial coordinates of each point. To fully utilize the continuity between time frames, multi-frame aggregation input is used. The point cloud data is first voxelized to divide the 3D point cloud data of each frame into voxels. The output shape of the voxelization becomes N×90×1. At the same time, each voxel is mapped to a high-dimensional embedding space by looking up the voxel dictionary.

[0052] Next, after passing through the embedding layer, the voxel feature data becomes N × 90 × 50 and is further flattened to N × 4500. This data is then input into a two-layer GRU encoder module. Each GRU unit in the encoder layer contains 512 hidden units, which are used to extract spatial and velocity features from the sequence. The encoding process outputs two types of states: a sequence of intermediate states (with a shape of N × 512) and a final state (with a shape of 1 × 512).

[0053] During the decoding phase, the decoder consists of a single GRU layer and an attention layer. The hidden state dimension of the GRU unit is also 512. The decoder first receives the final state of the encoder as its initial input to capture global features. Simultaneously, during each decoding step, the attention layer focuses on key features of the input data through a sequence of intermediate states and outputs the current decoder state. The decoder output has a shape of 1×1024 and is concatenated with the prediction result of the previous time step to dynamically adjust the prediction.

[0054] After the decoder output, the concatenated feature maps are further processed through a fully connected layer, and the output layer is mapped to the final predicted voxel format. Finally, a devoxelization operation is performed using the voxel dictionary to restore the predicted voxels to 3D spatial coordinates, obtaining skeleton joint positions with a resolution of 25×3, achieving a coarse human pose estimation.

[0055] The spatial velocity feature fine-tuning network model is built based on 3D convolution and long short-term memory networks, and is used to extract features from velocity data projected in the point cloud voxel space to obtain spatial velocity features. It includes:

[0056] a. Project the velocity into the point cloud voxel space to obtain the point cloud velocity feature;

[0057] b. Extract features from the 3D point cloud spatial data through 3D convolution to obtain spatial velocity features;

[0058] c. Input the spatial velocity features into the LSTM layer to capture the dependencies between consecutive frames of the time series velocity data;

[0059] d. Use a fully connected layer to align the feature channels;

[0060] e. Use a fully connected layer to fine-tune the output of the rough estimate network.

[0061] Build a fine-tuning network model based on convolution and timing models, such as Figure 2 As shown in Figure 3, the fine-tuning network includes a feature extraction module and a fine feature fusion module. This network focuses on fine-tuning the pose information output by the coarse estimation network.

[0062] First, the input of the fine-tuning network is the velocity data projected in the voxelized space of the point cloud. The velocity information is associated with the position of each point in the point cloud, that is, the velocity information of the point cloud point. It is obtained through the Doppler effect of the radar. The frequency change of the radar wave echo is proportional to the speed of the object. Therefore, by analyzing the frequency change of the echo signal, that is, the fast Fourier transform (FFT), the speed of the object under the radar perspective can be estimated. The shape is x∈R N×35×29×45 , N is the number of time frames, and 35, 29, and 45 are the spatial dimensions of the point cloud. This input data is processed by a 3D convolutional layer to extract features in the spatial and velocity dimensions. The feature extraction module contains multiple convolutional layers, each with a different number of 3D convolution kernels to extract multi-scale features.

[0063] In the feature extraction module, the input data is first channel-expanded, transformed into a four-dimensional tensor (35, 29, 45, 1), and then fed into the first 3D convolutional layer. This layer has 64 3×3×3 convolution kernels, uses the ReLU activation function, and has 64 output channels. It then passes through a 2×2×2 max pooling layer. The second 3D convolutional layer has 128 3×3×3 convolution kernels, also uses the ReLU activation function, and has 128 output channels. To enhance network robustness, a dropout layer is added after the second 3D convolutional layer, maintaining a probability of 0.5. Finally, it passes through another 2×2×2 max pooling layer.

[0064] After the convolution and pooling operations are completed, the feature map is mapped to the final channel dimension through the GlobalAveragePooling3D layer. The feature vector then enters the Long Short-Term Memory (LSTM) layer. The LSTM unit has 512 hidden nodes to capture the temporal dependencies in the input data. The feature is then input into a fully connected layer and mapped to the same dimension as the output of the main branch of the coarse estimation network (i.e., 2602) through the ReLU activation function, forming a finely tuned radar velocity feature vector.

[0065] S3. Fine-tune the rough estimate of the human skeleton point position output by the rough estimate network branch model based on the spatial velocity feature output by the spatial velocity feature fine-tuning network model, thereby generating an accurate human skeleton point position.

[0066] In this step, the human body posture skeleton point positions obtained by the rough estimation network are further fine-tuned through the spatial velocity features through a fine-tuning network model based on 3D convolution and LSTM.

[0067] Radar velocity feature vector F generated by fine-tuning the network v It is adjusted by a scaling factor α (e.g. 0.1) to ensure a moderate impact on the main branch output.

[0068] F v ′ =α·F v

[0069] Then, it is spliced ​​with the output of the main branch, and the coordinate information and feature information are merged column by column to obtain the fused features:

[0070] C fused =[C main ∣F v ′ ]

[0071] Among them, the splicing operation “|” means merging by columns in the dimension, and the main branch C main The coordinate dimension of the output is (N, D), the feature dimension of the fine-tuning branch output is (N, M), and the fused feature vector C fused The dimension is (N, D+M).

[0072] The final fusion features are directly passed to the output layer, and the fine-tuned human posture prediction results are obtained through the fully connected layer.

[0073] The present invention also discloses that the training steps of the rough estimation network branch model and the spatial velocity feature fine-tuning network model include:

[0074] S01, obtain human body posture data collected by millimeter wave radar, perform Fourier transform, MTI filtering and CFAR detection on the human body posture data, so as to obtain human body radar point cloud data. Specifically, Figure 3 Figure 2 illustrates the preprocessing process for millimeter-wave radar data. First, the radar echo signal is processed using Fast-Time FFT to obtain target range information. Subsequently, Slow-Time FFT is used to analyze frequency shifts and extract the target's radial velocity information. Next, MTI filtering is applied to remove static clutter, ensuring the detection of dynamic targets. Target detection is performed using the CFAR algorithm to eliminate background noise and ensure accurate identification of valid targets. Finally, Channel FFT is used to obtain target angle information from the multi-channel signal. Coordinate transformation is then performed, converting polar coordinates into three-dimensional coordinates in a Cartesian coordinate system. Ultimately, accurate point cloud data is generated, providing the basis for subsequent pose estimation and target analysis.

[0075] S02, obtaining human joint position data corresponding to the human body radar point cloud data through a Kinect camera;

[0076] S03. Using the human body radar point cloud data as input data and the human body joint position data as output data, the rough estimation network branch model and the spatial velocity feature fine-tuning network model are trained.

[0077] The present invention improves the accuracy and robustness of posture estimation through a multi-branch network structure, combining the advantages of a coarse estimation network and a fine-tuning network. The coarse estimation network uses modules such as GRU and attention layer to efficiently extract and model the spatiotemporal features of point cloud data, thereby achieving a rough estimate of the preliminary coordinates. The fine-tuning network effectively captures spatial velocity features through a feature extraction module composed of modules such as 3D convolutional layers and pooling layers, and further optimizes the output of the coarse estimation network. The two network branches work together to effectively improve the accuracy and consistency of posture estimation. At the same time, the multi-frame aggregate input makes full use of the complementarity between frames, and retains the radar point cloud data of the previous two frames and the current frame through a sliding window mechanism to form a multi-frame aggregate data input network. It effectively utilizes the continuity features between frames, improves the continuity of movements when predicting human movements, and further enhances the model's dynamic tracking capability for human posture.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating human body posture from millimeter-wave radar point cloud based on velocity correction, characterized in that: The following steps are involved: Obtain the radar point cloud data to be estimated, perform multi-frame point cloud aggregation on the radar point cloud data in the form of a sliding window, and thus generate radar point cloud input data; The radar point cloud input data is respectively input into the trained coarse estimation network branch model and the spatial velocity feature fine-tuning network model; the coarse estimation network branch model is constructed based on the human sequence codec structure and is used to roughly estimate the position of human skeletal points. The coarse estimation network branch model includes a point cloud voxelization module, two GRU encoding units, an attention layer and a GRU decoder; the spatial velocity feature fine-tuning network model is constructed based on 3D convolution and long short-term memory network, and is used to extract features from the velocity data projected in the point cloud voxelization space to obtain spatial velocity features; The rough estimated human skeleton point positions output by the rough estimation network branch model are fine-tuned based on the spatial velocity features output by the spatial velocity feature fine-tuning network model, thereby generating accurate human skeleton point positions.

2. The method for estimating human body posture from millimeter-wave radar point cloud based on velocity correction according to claim 1, characterized in that: The position of human skeleton points is roughly estimated based on the rough estimation network branch model, including: Perform voxel mapping encoding on the radar point cloud input data to obtain the sequence corresponding to the point cloud position; Learning the encoding representation of the sequence through the GRU encoding unit; The attention layer allows the learning process to focus on the final encoder state while also focusing on the intermediate encoder state, thereby improving the efficiency of the decoder. The decoder predicts the final human skeleton point sequence encoding representation; The human skeleton encoding sequence output by the decoder is devoxelized to obtain a rough estimate of the human skeleton point position.

3. The method for estimating human body posture from millimeter-wave radar point cloud based on velocity correction according to claim 1, characterized in that: Based on the spatial velocity feature, the fine-tuned network model is used to extract features from the radar point cloud data after splicing with the velocity feature, including: Acquire velocity data based on radar point cloud input data, and project the velocity data into point cloud voxelized space to acquire point cloud velocity data; Perform feature extraction on the point cloud velocity data through 3D convolution to obtain spatial velocity features; The spatial velocity features are input into the LSTM layer to capture the dependencies between consecutive frames of the time series velocity data.

4. The method for estimating human body posture from millimeter-wave radar point cloud based on velocity correction according to claim 1, characterized in that: Fine-tuning the rough estimated human skeleton point position output by the rough estimation network branch model based on the spatial velocity feature output by the spatial velocity feature fine-tuning network model includes: Use a fully connected layer to align the spatial velocity features with their channels; A fully connected layer is used to fine-tune the rough estimate of the human skeleton point position output by the rough estimation network.

5. The method for estimating human body posture from millimeter-wave radar point cloud based on velocity correction according to claim 1, characterized in that: Aggregate multiple frames of radar point cloud data in the form of a sliding window to generate radar point cloud input data, including: In the form of a sliding window, the previous two frames of radar point cloud data and the current frame data are retained each time to fully utilize the continuity features between frames.

6. The method for estimating human body posture from millimeter-wave radar point cloud based on velocity correction according to claim 1, characterized in that: The training steps of the rough estimation network branch model and the spatial velocity feature fine-tuning network model include: Acquire human body posture data collected by millimeter wave radar, perform Fourier transform, MTI filtering and CFAR detection on the human body posture data, thereby obtaining human body radar point cloud data; Acquire human joint point position data corresponding to the human body radar point cloud data through a Kinect camera; The human body radar point cloud data is used as input data and the human body joint point position data is used as output data to train the rough estimation network branch model and the spatial velocity feature fine-tuning network model.

Images