First-person pose estimation method and system based on head-mounted millimeter wave radar

Abstract

The application discloses a first-person pose estimation method and system based on a head-mounted millimeter wave radar, first, the motion data of the radar is obtained in the IMU integrated with the radar, the motion data is sent into a neural network for learning, and the relative 6D trajectory of the radar is obtained; then, the millimeter wave point cloud is all converted from the radar coordinate system into the H coordinate system, the influence of the radar motion is decoupled, the accurate joint position of the upper body is learned through the neural network; finally, the correlation between the upper body and the lower body is utilized, the feature extraction is respectively performed on the upper body skeleton and the lower body point cloud through the neural network, the feature fusion is performed, the human motion priori is learned, the accurate joint position of the lower body is obtained, and the whole body human pose estimation is finally completed. The method realizes the pose estimation of the lower body of the human body by using the human motion priori, overcomes the challenge of the missing information of the lower body, realizes the robust human pose estimation in various environments, avoids the privacy leakage, and is more accurate and safe.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of wireless sensing, and mainly relates to a first-person pose estimation method and system based on a head-mounted millimeter wave radar. BACKGROUND

[0002] Human pose estimation aims to reconstruct three-dimensional body motion and plays an important role in many application fields, such as sports analysis, human-computer interaction, augmented reality (AR), virtual reality (VR) and medical rehabilitation. Traditional human pose is captured by motion capture devices (such as cameras) installed around the scene or a large number of inertial measurement units (IMUs) worn on the body. However, the fixed motion capture device infrastructure is limited by a limited sensing range, which limits the space of human motion, so it cannot meet the daily activities involving large-scale movement. On the other hand, the IMU worn on the body requires tedious setup and complex calibration operations, which brings inconvenience to the user and hinders normal activities and social interactions.

[0003] In recent years, the popularity of head-mounted devices (such as VR glasses and smart helmets) has inspired a new direction of first-person pose estimation. Specifically, it estimates the pose through a single head-mounted device worn by the user, providing mobility and convenience for the user. For example, Apple Vision Pro is equipped with multiple cameras to replace the traditional handle. However, vision-based methods are very sensitive to lighting conditions, smoke, dust and human appearance, and often raise privacy issues.

[0004] The emerging integrated millimeter wave radar has low cost and miniaturization (about 10 centimeters), so it is easy to embed into head-mounted devices (such as AR helmets). In addition, recent applications have demonstrated the potential of radar in providing various human sensing capabilities, while being robust to poor lighting or weather, protecting privacy and being non-intrusive to users. First-person pose estimation based on radar can achieve a wide range of applications, such as immersive virtual reality, motion-assisted analysis, AR visual enhancement for first aid personnel, and safety behavior detection for drivers.

[0005] Although radar has recently achieved success in various human sensing tasks, the first-person perspective poses two unique challenges for pose estimation. First, the head-mounted device is non-stationary, so the radar signal not only contains changes in pose, but also includes random device movement caused by head motion. The existing state-of-the-art pose estimation methods cannot capture stable spatio-temporal features, resulting in significant joint localization errors (up to 16 centimeters at maximum). Second, due to the top-down perspective, the radar signal is severely affected by mirror reflection and upper limb self-occlusion in the lower body, resulting in the radar being able to sense only a small amount of lower limb movement, so it is challenging to accurately estimate the lower body. SUMMARY

[0006] The present application is just aimed at the problem of large error and missing lower body information in the prior art pose estimation method, and provides a first-person pose estimation method and system based on a head-mounted millimeter wave radar. First, radar trajectory tracking based on head movement features is performed. The movement data of the radar is obtained from the IMU integrated with the radar, the movement data is sent into a neural network for learning, and the relative 6D trajectory of the radar is obtained. Then, the millimeter wave point cloud is converted from the radar coordinate system to the H coordinate system, the influence of the radar movement is decoupled, and the accurate joint position of the upper body is learned through the neural network. Finally, the correlation between the upper body and the lower body is utilized, the feature extraction of the upper body skeleton and the lower body point cloud is performed through the neural network, the feature fusion is performed, the human motion prior is learned, the accurate joint position of the lower body is obtained, and the whole body pose estimation is finally completed. The present method realizes the pose estimation of the lower body by using the human motion prior, overcomes the challenge of missing lower body information, realizes the robust human pose estimation in various environments and avoids privacy leakage, and is more accurate and safe.

[0007] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is: a first-person pose estimation method based on a head-mounted millimeter wave radar, comprising the following steps:

[0008] S1, radar trajectory tracking based on head movement features: the movement data of the radar is obtained from the IMU integrated with the radar, the movement data is sent into a neural network for learning, and the 6D trajectory of the radar relative to the H point is obtained; the H point is the middle position of the human clavicle;

[0009] S2, upper body pose estimation based on radar movement decoupling: the millimeter wave point cloud is converted from the radar coordinate system to the H coordinate system, the influence of the radar movement is decoupled, and the accurate joint position of the upper body is learned through the neural network;

[0010] S3, lower body pose estimation based on human motion prior: the correlation between the upper body and the lower body is utilized, the feature extraction of the upper body skeleton and the lower body point cloud is performed through the neural network, the feature fusion is performed, the human motion prior is learned, and the accurate joint position of the lower body is obtained.

[0011] As an improvement of the present application, the step S1 specifically comprises:

[0012] S11: the movement data of the radar is measured by the IMU integrated with the radar, wherein the t-th frame includes 3D acceleration t , 3D angular velocity t As input, it is mapped to a 512-dimensional feature vector through a (6, 512) full connection layer (FC);

[0013] S12: Extracting micro-motion features using bidirectional two-layer long short-term memory neural network (LSTM) on 20 frames of feature sequences, and the output feature dimension is 1024;

[0014] S13: Aggregating the features of each time step through a self-attention mechanism layer with a size of (1024, 1) to obtain the feature representation of the 20-frame sequence;

[0015] S14: Inputting the aggregated features into a LSTM with the same parameters to extract intense motion features, and the output feature dimension is also 1024;

[0016] S15: Mapping the features to the motion trajectory of the radar relative to point H through a fully connected layer with a size of (1024, 9), i.e., a six-dimensional vector representing the orientation and a three-dimensional vector representing the position Point H represents the middle position of the clavicle of the human body;

[0017] S16: Calculating the orientation and position errors of the predicted orientation and position using the following loss function, calculating the gradient and performing back propagation, and updating the network parameters until the network converges:

[0018]

[0019] where α and β are weight coefficients, R t is the real radar orientation, p t is the real position of the radar, and T1 is the length of the point cloud frame sequence.

[0020] As an improvement of the present application, the step S2 specifically comprises:

[0021] S21: Converting the point cloud coordinates obtained by the millimeter wave radar reflecting the human body to the H coordinate system based on the radar relative orientation R and position p obtained in step S1, wherein each point is represented by a six-dimensional vector including xyz three-dimensional coordinates, distance r, velocity v, and intensity s;

[0022] S22: Inputting the point cloud into a shared multi-layer perception (shared-MLP) with shared parameters for feature extraction, with a size of (6, 16, 32, 64), and using batch normalization (BN) and ReLU activation function after each layer;

[0023] S23: Using a self-attention mechanism with a size of (64, 1) to adaptively learn the weight of each point and aggregate the features of different points to obtain a high-dimensional feature representation at the frame level;

[0024] S24: Inputting the frame-level feature representation into a three-layer bidirectional LSTM to extract time sequence features, and the output feature is 128-dimensional;

[0025] S25: A multi-layer perception (MLP) with size (128, 64, 42) maps the time series features to the rotation angles of the upper body joints relative to their parent joints RJ i where i = 1, 2, …, 14, RJ i is represented by a six-dimensional vector;

[0026] S26: Convert the six-dimensional rotation representation to a rotation matrix representation using Schmidt orthogonalization, apply human forward kinematics to get the human upper body joint positions from the joint rotation matrix is represented by a three-dimensional vector with its xyz three-dimensional coordinates;

[0027] S27: Calculate the predicted joint position error using the following loss function, calculate the gradient and perform backpropagation, update the parameters until the network converges;

[0028]

[0029] where is the true upper body joint position.

[0030] As another improvement of the application, the upper body joints include left and right wrists, left and right elbows, left and right shoulders, left and right clavicles, head, neck, chest, navel, pelvis and left and right hip joints, and the neck is taken as the root joint.

[0031] As another improvement of the application, the step S3 specifically includes:

[0032] S31: The human upper body joint positions obtained in step S2 are sent to a graph convolutional neural network (GCN) with three layers with sizes (32, 64, 128) respectively to extract features, and the upper body pose features H are obtained;

[0033] S32: The millimeter wave point cloud is cropped with the x-axis coordinate of the pelvis joint as the boundary to obtain the lower body point cloud, and the insufficient 64 is zero-padded, and sent to a multi-layer perception with shared parameters with size (6, 32, 64, 128) to obtain the lower body point cloud features Z;

[0034] S33: The point cloud features Z and the skeleton features H are fused using a mutual attention mechanism with size (128, 128) and residual connection;

[0035] S34: Use a self-attention mechanism with (128, 1) to aggregate into frame-level features;

[0036] S35: The frame-level features are sent to a three-layer bidirectional LSTM to extract time series features;

[0037] S36: Map the time series features to six lower body joint rotation angles using a multi-layer perceptron with size (128, 64, 36), also represented as a six-dimensional vector;

[0038] S37: Convert the six-dimensional rotation representation to a rotation matrix representation using Schmidt orthogonalization, apply forward kinematics to get lower body joint positions

[0039] S38: Calculate the error of predicted joint positions using the following loss function, calculate the gradient and backpropagate, update the parameters until the network converges;

[0040]

[0041] where is the true lower body joint position.

[0042] As another improvement of the present application, the lower body joints include left and right feet, left and right ankles, left and right knees, and left and right hip joints, wherein the left and right hips are two root joint positions.

[0043] As another improvement of the present application, the forward kinematics calculation in step S26 and step S37 is as follows:

[0044]

[0045] where J parent(i) represents the parent node of J i in the skeleton structure, and represent the initial positions of J i and J parent(i) respectively.

[0046] To achieve the above purpose, the technical scheme adopted by the present application is: a first-person pose estimation system based on a head-mounted millimeter wave radar, at least including a radar trajectory tracking module, an upper body pose estimation module and a lower body pose estimation module,

[0047] The radar trajectory tracking module: using the IMU integrated with the radar, based on the radar trajectory tracking network of head movement features, the accurate radar relative 6D trajectory is trained, the radar trajectory tracking network captures the human head movement law through LSTM of different time scales and self-attention fusion mechanism at time steps;

[0048] The upper body posture estimation module is based on a radar posture decoupling upper body posture estimation network, and an upper body posture is trained; input of the module is millimeter wave point cloud converted to an H coordinate system, semantic features of different points are extracted through a shared parameter MLP in PointNet, frame level features are aggregated through a self-attention mechanism, and the hidden layer features are mapped to the human body upper body posture through an MLP and a forward kinematics module.

[0049] The lower body posture estimation module is based on human motion prior, the upper body posture and the lower body point cloud are fused to estimate the lower body posture, so as to complete the whole body human posture estimation.

[0050] As another improvement of the present application, the input of the lower body posture estimation module is the upper body posture output by the upper body posture estimation module, and the lower body point cloud clipped through the pelvic point; after the features are extracted by the corresponding feature extraction network respectively, the feature fusion is performed on the sparse point cloud in the semantic context of the upper body posture, more accurate lower body features are obtained, the hidden layer features are mapped to the human body lower body posture through aggregation and time sequence feature extraction, and the whole body human posture is obtained after the upper body posture and the lower body posture are spliced.

[0051] Compared with the prior art, the present application has the beneficial effects:

[0052] (1) The first person posture estimation method based on the head-mounted millimeter wave radar can realize robust human posture estimation in various environments without limitation of activity range, compared with the traditional camera which depends on light, exposes privacy and is easily affected by bad weather, the method only uses low-cost radar and IMU equipment, does not depend on light and is not affected by smoke dust and other bad conditions, is easier to implement, and avoids privacy problems.

[0053] (2) The radar trajectory tracking method based on head motion features in step S1 avoids the drift problem of IMU position measurement, and obtains accurate 6D radar trajectory.

[0054] (3) The lower body posture estimation based on human motion prior in step S3 can realize accurate lower body posture estimation effect in the case of lack of lower body information, and the result is more accurate. DETAILED DESCRIPTION

[0055] Figure 1 is a step flowchart of the method of the present application;

[0056] Figure 2 is a radar posture diagram relative to the neck joint (H point) in step S1 of the method of the present application;

[0057] Figure 3 is the working principle diagram of the radar trajectory tracking module in the system of the present application;

[0058] Figure 4 is the working principle diagram of the upper body posture estimation module (UpperNet) in the system of the present application;

[0059] Figure 5 is the working principle diagram of the lower body posture estimation module (LowerNet) in the system of the present application;

[0060] Figure 6 is the prototype diagram of the system experimental equipment in the test example of the present application. DETAILED DESCRIPTION

[0061] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0062] Example 1

[0063] The first-person posture estimation system based on head-mounted millimeter wave radar at least includes a radar trajectory tracking module, an upper body posture estimation module and a lower body posture estimation module,

[0064] wherein the radar trajectory tracking module is as shown in Figure 3 , an IMU integrated with the radar is used, a radar trajectory tracking network based on head movement features is used, and an accurate radar relative 6D trajectory is trained. The radar relative trajectory is a trajectory relative to the human neck joint (represented by point H) as shown in Figure 2 The radar trajectory tracking network captures the human head movement law through LSTM of different time scales and self-attention fusion mechanism at time steps;

[0065] The upper body posture estimation module is as shown in Figure 4 , an upper body posture estimation network based on radar posture decoupling is used, and the upper body posture is trained. The input of this module is the millimeter wave point cloud converted to the H coordinate system, the semantic features of different points are extracted through the MLP with shared parameters in PointNet, then the frame-level features are aggregated through the self-attention mechanism, the connection of the point clouds between different frames is captured through the LSTM, and finally the hidden layer features are mapped to the human upper body posture through the MLP and the forward kinematics module;

[0066] The lower body posture estimation module is as shown in Figure 5As shown, based on the human motion prior, the upper body pose and the lower body point cloud are fused to estimate the lower body pose, thereby completing the full-body human pose estimation. The input of this module contains two parts, one is the upper body pose output by the upper body pose module, and the other is the lower body point cloud cropped by the pelvis point. Since the two inputs share a coordinate space, the corresponding feature extraction network is used to extract features, and then the sparse point cloud is fused in the semantic context of the upper body pose, so as to obtain more accurate lower body features. Subsequently, by aggregating and extracting the temporal features, the hidden layer features are mapped to the human lower body pose, and the complete human pose is obtained after splicing with the upper body pose.

[0067] Firstly, the system uses the IMU integrated with the radar, the radar trajectory tracking network based on head motion features, to train the accurate radar relative 6D trajectory; secondly, based on the upper body pose estimation network decoupled from the radar pose, a more accurate upper body pose is trained; finally, based on the human motion prior, the upper body pose and the lower body point cloud are fused to estimate the lower body pose, thereby completing the full-body human pose estimation. The system uses low-cost radar and IMU devices, overcomes the limitations of traditional cameras relying on light, exposing privacy, and being easily affected by bad weather, is easy to implement, and at the same time overcomes the problem of missing lower body information, and the pose estimation is more accurate and safe.

[0068] Embodiment 2

[0069] The first-person pose estimation method based on the head-mounted millimeter wave radar, as shown in Figure 1 , realizes stable first-person pose estimation through radar trajectory tracking and human motion prior, and specifically includes the following steps:

[0070] S1, radar trajectory tracking based on head motion features: obtain the motion data of the radar in the IMU integrated with the radar, input the motion data into the neural network after learning, and obtain the 6D trajectory of the radar relative to the H point;

[0071] Wherein, the H point is the middle position of the human clavicle, as shown in Figure 2 ; the neural network model for realizing radar trajectory tracking is as shown in Figure 3 . Indicates the IMU data from t1 to t2, the IMU data of each time is acceleration α t and angular velocity ω t Spliced into a six-dimensional vector; δ and T2 are the selected short time interval and long time interval respectively; T1 is the point cloud frame time interval length; R r and P r Indicate the orientation and position of the radar; the step specifically includes:

[0072] S11: ximu mapping it to a 512-dimensional feature vector through a fully connected layer of size (6, 512);

[0073] S12: Extracting the micro-motion feature using bidirectional two-layer LSTM on the delta frame feature sequence, and the output feature dimension is 1024;

[0074] S13: Then, the feature of each time step is aggregated through a self-attention mechanism layer with a size of (1024, 1) to obtain the feature representation of the 20-frame sequence;

[0075] S14: The multiple aggregated features are input into the LSTM with the same parameters to extract the intense motion feature, and the output feature dimension is also 1024;

[0076] S15: The feature is mapped to the motion trajectory of the radar relative to the H point through a fully connected layer of size (1024, 9), i.e., a six-dimensional vector representing the orientation and a three-dimensional vector representing the position The H point represents the middle position of the human clavicle;

[0077] S16: The orientation and position errors predicted by the following loss function are calculated, the gradient is calculated and backpropagation is performed, and the network parameters are updated until the network converges:

[0078]

[0079] S2, upper body posture estimation based on radar motion decoupling: converting the millimeter wave point cloud from the radar coordinate system to the H coordinate system, decoupling the influence of the radar motion, and learning the accurate joint position of the upper body through a neural network;

[0080] After obtaining the radar posture, the model process in the coordinate space is converted to the coordinate space with the H point as the root node, specifically, the millimeter wave point cloud and the human body posture are converted to the H coordinate system, and then the neural network model as shown in Figure 4 is used to obtain the upper body posture, and the steps specifically include:

[0081] S21: Based on the radar relative rotation R and translation t obtained in step S1, the point cloud coordinates obtained by reflecting the human body by the millimeter wave radar are converted to the H coordinate system, wherein each point is represented by a six-dimensional vector, including xyz three-dimensional coordinates, distance r, velocity v and intensity s; the point cloud with less than 128 points per frame is zero-filled;

[0082] S22: The point cloud is input into the multi-layer perception machine with shared parameters for feature extraction, with a size of (6, 16, 32, 64), and batch normalization (BN) and ReLU activation function are used after each layer;

[0083] S23: Then use the self-attention mechanism of (64, 1) to adaptively learn the weight of each point, thereby aggregating the features of different points, and obtaining the high-dimensional feature representation at the frame level;

[0084] S24: The frame-level feature representation is input into a three-layer bidirectional LSTM to extract time sequence features, and the output feature is 128-dimensional;

[0085] S25: A multilayer perceptron with a size of (128, 64, 42) maps the time sequence features to the rotation angles RJ of the fourteen joints of the upper body relative to their parent joints i , where i = 1, 2, …, 14, RJ i is represented by a six-dimensional vector;

[0086] S26: Convert the six-dimensional rotation representation to a rotation matrix representation using Schmidt orthogonalization, and apply human forward kinematics to obtain the joint positions of the upper body from the joint rotation matrix , which is represented by a three-dimensional vector with xyz three-dimensional coordinates; where the forward kinematics is calculated as follows:

[0087]

[0088] S27: Calculate the error of the predicted joint position using the following loss function, calculate the gradient and perform back propagation, and update the parameters until the network converges;

[0089]

[0090] S3, lower body pose estimation based on human motion prior: by using the correlation between the upper body and the lower body, the neural network is used to extract features from the upper body skeleton and the lower body point cloud respectively, and the features are fused, and the human motion prior is learned, so as to obtain the accurate joint position of the lower body of the human body.

[0091] Using the radar pose decoupled upper body pose and millimeter wave point cloud extracted in step S2 and step S1, estimate the human lower body pose using a deep neural network model as shown in Figure 5 , the steps specifically include:

[0092] S31: Map the human upper body joint position obtained in step S2, each joint as a node of the graph, and the limb as an edge, input into a three-layer graph convolutional neural network (GCN) with sizes of (32, 64, 128) respectively to extract features, and obtain the upper body pose feature H;

[0093] S32: The millimeter wave point cloud is cropped by taking the x-axis coordinate of the pelvis joint as the boundary to obtain a lower body point cloud, and the point cloud feature Z is obtained by inputting the lower body point cloud into a multi-layer perception network with shared parameters of size (6, 32, 64, 128).

[0094] S33: The point cloud feature Z and the skeleton feature H are fused by using a mutual attention mechanism with a size of (128, 128) and a residual connection, wherein Z is used as a query and H is used as a key value, and the specific calculation method is as follows:

[0095]

[0096] wherein W Q , W K , W V are learnable parameters with a dimension of (128, 128), and d' is 128.

[0097] S34: A self-attention mechanism with a size of (128, 1) is used for aggregation to obtain a frame-level feature.

[0098] S35: The frame-level feature is input into a three-layer bidirectional LSTM to extract a time sequence feature.

[0099] S36: A multi-layer perception network with a size of (128, 64, 36) is used to map the time sequence feature to six lower body joint rotation angles, which are also represented by a six-dimensional vector.

[0100] S37: The six-dimensional rotation representation is converted into a rotation matrix representation by using Schmidt orthogonalization, and forward kinematics is applied to obtain the lower body joint positions

[0101] S38: The predicted joint position error is calculated by using the following loss function, the gradient is calculated and back propagation is performed, and the parameters are updated until the network converges.

[0102]

[0103] The present application uses a data-driven method to learn the head movement rule and body movement rule of a human body by using a deep neural network, overcomes the challenge caused by radar movement and sparse lower body information, and thus realizes accurate first-person pose estimation. Conceptually, the present application is a pose estimation design that provides high mobility, environmental robustness and privacy protection, and is more accurate and safe.

[0104] Test example

[0105] In order to verify the effectiveness of the method proposed in the present application, a human pose estimation experiment is carried out in an indoor environment to show the performance advantages and influences of the present application.

[0106] The experiment is performed with the following configuration:

[0107] Hardware setup: 1. Millimeter wave radar IWR6843 ISK-ODS, used to collect millimeter wave point cloud data;

[0108] 2. IMU Wheeltec N100, used to collect IMU data;

[0109] 3. Azure Kinect v2, used to collect human real pose, train network;

[0110] 4. 3D printed helmet with checkerboard, used to install and calibrate radar and IMU.

[0111] Experiment design:

[0112] 1. The experimenter is required to wear the experimental helmet as shown in Figure 6 and perform the specified action;

[0113] 2. The actions include the following thirteen daily actions: walking in place; walking; arm stretching; shaking head; nodding; turning head; looking left and right while walking in place; looking up and down while walking in place; looking up and down while waving arms; looking left and right while waving arms; bowing; high leg lifting; squatting;

[0114] 3. The average joint position error (denoted by S) and the average rotation angle error (Q) are used to measure the accuracy of the pose estimation effect. Where S is the average Euclidean distance between the predicted joint position and the real position, and Q is the average angle difference between the predicted joint rotation angle and the real joint rotation angle;

[0115] 4. Select mmMesh, P4Transformer, PCB, mmPose-NLP as comparison methods, where mmMesh is the best existing method.

[0116] The experimental results are shown in the following table:

[0117]

[0118] As can be seen from the table, the estimation accuracy of the whole body, upper body and lower body is included. The scheme achieves an average accuracy of 4.3 cm and 4.9° in whole body pose estimation, an average accuracy of 4.2 cm and 5.2° in upper body, and an average accuracy of 5.4 cm and 4.2° in lower body. Compared with the best existing method (mmMesh), the system improves the accuracy by 44.2%.

[0119] As can be seen from the above experiment or data, in human daily actions, the present application can achieve an average joint positioning error of 4.3 cm and a joint rotation angle error of 4.9°.

[0120] To sum up, the present application provides a first-person pose estimation method and system based on a head-mounted millimeter wave radar. The method uses human motion priors to estimate the pose of the lower body, overcoming the challenge of missing information about the lower body, enabling robust human pose estimation in various environments, and being more accurate and safe. Compared to traditional cameras, which rely on light, expose privacy, and are easily affected by bad weather, the present application uses low-cost radar and IMU devices, does not rely on light, and is unaffected by bad conditions such as smoke and dust, making it easier to implement and avoiding privacy issues.

[0121] It should be noted that the above content only illustrates the technical idea of the present application and cannot limit the protection scope of the present application. For ordinary skilled persons in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which fall within the scope of protection of the claims of the present application.

Claims

1. A first-person attitude estimation method based on head-mounted millimeter-wave radar, characterized in that, Includes the following steps: S1, Radar trajectory tracking based on head movement characteristics: The radar motion data is obtained through the IMU integrated with the radar, and the motion data is fed into the neural network for learning to obtain the 6D trajectory of the radar relative to point H; the point H is the middle position of the human clavicle. S11: Motion data of the radar is obtained by measuring with the IMU integrated with the radar, where the t-th frame includes 3D acceleration. 3D angular velocity As input, it is mapped to a 512-dimensional feature vector through a (6, 512) fully connected layer (FC); S12: Extract minute motion features using a bidirectional two-layer Long Short-Term Memory (LSTM) neural network on a 20-frame feature sequence, with an output feature dimension of 1024. S13: The features of each time step are aggregated through a self-attention mechanism layer of size (1024, 1) to obtain the feature representation of the 20-frame sequence; S14: Multiple aggregated features are input into an LSTM with the same parameters to extract violent motion features, and the output feature dimension is also 1024. S15: The features are mapped to the radar's 6D motion trajectory relative to point H through a (1024, 9) fully connected layer, i.e., the relative orientation represented by a six-dimensional vector. Relative position represented by a three-dimensional vector Point H represents the middle position of the human clavicle; S2, Upper body posture estimation based on radar motion decoupling: The millimeter-wave point cloud is completely transformed from the radar coordinate system to the H coordinate system to decouple the influence of radar motion, and the accurate joint positions of the upper body are learned through neural networks. Based on the radar relative orientation R and position t obtained in step S1, the point cloud coordinates obtained by the millimeter-wave radar reflecting off the human body are transformed into the H coordinate system, where each point is represented by a six-dimensional vector, including three-dimensional coordinates xyz, distance r, velocity v, and intensity s; the point cloud is input into a multilayer perceptron for feature extraction, the weight of each point is adaptively learned based on a self-attention mechanism, time series features are extracted based on LSTM, and mapped to the rotation angle of the upper body joints relative to their parent joints, the rotation angle is represented by a six-dimensional vector, and the position of the upper body joints is obtained by applying human forward kinematics; S3, Lower body pose estimation based on human motion prior: Utilizing the relationship between the upper and lower body, features are extracted from the upper body skeleton and lower body point cloud through a neural network, and then the features are fused to learn human motion prior, thereby obtaining the accurate joint positions of the lower body. The upper body posture features are obtained by taking the joint positions of the upper body obtained in step S2. The lower body point cloud features are obtained by cropping millimeter-wave point clouds and using a multilayer perceptron. The upper body pose features and lower body point cloud features are fused and aggregated into frame-level features through a self-attention mechanism. Time series features are extracted by LSTM and mapped to the lower body joint rotation angles. The lower body joint rotation angles are represented by a six-dimensional vector, and the lower body joint positions are obtained by applying forward kinematics.

2. The first-person attitude estimation method based on head-mounted millimeter-wave radar as described in claim 1, characterized in that: Step S1 further includes: S16: Calculate the predicted orientation and position errors using the following loss function, calculate the gradient and perform backpropagation, and update the network parameters until the network converges: ; in These are the weighting coefficients. For the actual radar orientation, For the radar's actual location, The length of the point cloud frame sequence.

3. The first-person attitude estimation method based on head-mounted millimeter-wave radar as described in claim 2, characterized in that: Step S2 specifically includes: S21: Based on the radar relative orientation R and position t obtained in step S1, the point cloud coordinates obtained by the millimeter-wave radar reflecting off the human body are transformed into the H coordinate system, where each point is represented by a six-dimensional vector, including the three-dimensional coordinates xyz, distance r, velocity v, and intensity s; S22: Input the point cloud into a shared-MLP with shared parameters for feature extraction. The size of the shared-MLP is (6, 16, 32, 64). After each layer, batch normalization (BN) and ReLU activation function are used. S23: Use the (64,1) self-attention mechanism to adaptively learn the weight of each point, aggregate the features of different points, and obtain a frame-level high-dimensional feature representation; S24: Frame-level feature representation: Time series features are extracted from the input three-layer bidirectional LSTM, and the output features are 128-dimensional. S25: A multilayer perceptron (MLP) of size (128, 64, 42) maps time-series features to rotation angles of upper body joints relative to their parent joints. , where i = 1, 2, …, 14, Represented by a six-dimensional vector; S26: The six-dimensional rotation representation is converted into a rotation matrix representation using Schmidt orthogonalization. Forward kinematics is then applied to obtain the joint positions of the upper body from the joint rotation matrix. , Its xyz three-dimensional coordinates are represented by a three-dimensional vector; S27: Calculate the predicted joint position error using the following loss function, calculate the gradient and perform backpropagation, and update the parameters until the network converges; ; in The actual joint positions of the upper body are represented by 15, which represents the total number of joints.

4. The first-person attitude estimation method based on head-mounted millimeter-wave radar as described in claim 3, characterized in that: The upper body joints include the left and right wrists, left and right elbows, left and right shoulders, left and right collarbones, head, neck, chest, navel, pelvis and left and right hip joints, with the neck serving as the root joint.

5. The first-person attitude estimation method based on head-mounted millimeter-wave radar as described in claim 3, characterized in that: Step S3 specifically includes: S31: The joint positions of the upper body obtained in step S2 are fed into a graph convolutional neural network (GCN) with three layers of size (32, 64, 128) for feature extraction to obtain the upper body posture features H. S32: The millimeter-wave point cloud is cropped with the x-axis coordinates of the pelvic joints as the boundary to obtain the lower body point cloud. Points with less than 64 are padded with zeros and fed into a multilayer perceptron with shared parameters of size (6, 32, 64, 128) to obtain the lower body point cloud feature Z. S33: The point cloud features Z and the skeleton features H are fused using a mutual attention mechanism of size (128, 128) and residual connections; S34: Aggregate into frame-level features using a (128, 1) self-attention mechanism; S35: Feed the frame-level features into a three-layer bidirectional LSTM to extract time-series features; S36: A multilayer perceptron of size (128, 64, 36) is used to map time series features to six lower body joint rotation angles, also represented by a six-dimensional vector; S37: The six-dimensional rotation representation is converted into a rotation matrix representation using Schmidt orthogonalization, and the lower body joint positions are obtained by applying forward kinematics. ; S38: Calculate the predicted joint position error using the following loss function, calculate the gradient and perform backpropagation, and update the parameters until the network converges; ; in This represents the actual joint positions of the lower body.

6. The first-person attitude estimation method based on head-mounted millimeter-wave radar as described in claim 5, characterized in that: The lower body joints include the left and right feet, left and right ankles, left and right knees, and left and right hip joints, with the left and right hips being two root joints.

7. The first-person attitude estimation method based on head-mounted millimeter-wave radar as described in claim 5, characterized in that: The forward kinematics calculations in steps S26 and S37 are as follows: ; in Indicating in the skeleton structure The parent node, and They represent and The initial position.

8. A first-person attitude estimation system based on a head-mounted millimeter-wave radar, implementing the method as described in claim 1, characterized in that: It includes at least a radar trajectory tracking module, an upper body attitude estimation module, and a lower body attitude estimation module. The radar trajectory tracking module utilizes an IMU integrated with the radar and a radar trajectory tracking network based on head motion characteristics to train an accurate radar relative 6D trajectory. The radar trajectory tracking network captures the patterns of human head motion through LSTM at different time scales and a self-attention fusion mechanism at time steps. The upper body pose estimation module is trained using an upper body pose estimation network based on radar pose decoupling to obtain the upper body pose. The input of the module is a millimeter-wave point cloud transformed to the H coordinate system. Semantic features are extracted from different points through the MLP with shared parameters in PointNet. Then, frame-level features are aggregated through a self-attention mechanism, and LSTM is used to capture the relationship between point clouds in different frames. Finally, the hidden layer features are mapped to the human upper body pose through the MLP and forward kinematics module. The lower body posture estimation module: Based on human motion priors, the upper body posture and lower body point cloud are fused to estimate the lower body posture, thereby completing the full-body human posture estimation.

9. The first-person attitude estimation system based on head-mounted millimeter-wave radar as described in claim 8, characterized in that: The lower body pose estimation module takes as input the upper body pose output by the upper body pose module and the lower body point cloud cropped by pelvic points. After extracting features using corresponding feature extraction networks, the sparse point cloud is fused in the semantic context of the upper body pose to obtain more accurate lower body features. Then, through aggregation and temporal feature extraction, the hidden layer features are mapped to the human lower body pose, and concatenated with the upper body pose to obtain the full-body human pose.

Images