A dynamic millimeter wave radar point cloud human body analysis method based on joint learning

Abstract

The present application belongs to the technical field of Internet of Things perception, and designs a dynamic millimeter wave radar point cloud human body analysis method based on joint learning. Especially, the human body analysis based on millimeter wave perception. Current human-centered millimeter wave perception mostly focuses on action recognition, gesture estimation and other scenarios, but they usually cannot obtain semantic information of millimeter wave point cloud, that is, cannot distinguish body part information corresponding to each radar point, so a dynamic millimeter wave radar point cloud human body analysis scheme is needed. The content of the method includes: for millimeter wave point cloud data, firstly clustering, then using a multi-task learning model to jointly perform human body analysis and pose estimation tasks for feature extraction, and then performing multi-task feature fusion through a non-local network, and finally outputting the result as a point cloud with annotated semantic labels.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) sensing technology, and in particular to a dynamic millimeter-wave radar point cloud human body analysis method based on joint learning. Background Technology

[0002] Perceiving and understanding human activity plays an increasingly important role in human-centered intelligent applications. Traditional methods, employing cameras or body contact sensors, are susceptible to harsh environments and pose privacy concerns. In the area of ​​human perception, recent years have seen a surge in research using millimeter-wave radar, demonstrating its effectiveness in gesture and activity recognition, pose estimation, and identity recognition. However, these tasks typically cannot explicitly acquire semantic information from millimeter-wave point clouds, making it difficult to distinguish the body part information corresponding to each radar point and achieve human body analysis.

[0003] In human perception applications, fine-grained information about body parts is constantly needed, but the lack of such semantic information significantly limits millimeter-wave radar from becoming an enabling technology for human-centered computing in daily life. Meanwhile, using semantic information as an additional input channel can make human perception tasks more robust. Various computer vision tasks have demonstrated that including semantic information input can significantly improve the accuracy of pose estimation, activity recognition, and person recognition. This advantage is even more pronounced for millimeter-wave radar because millimeter-wave point clouds are inherently of lower image quality than those from visual sensors. Therefore, a technical solution is needed to achieve dynamic millimeter-wave radar point cloud human body analysis, obtaining point clouds with semantic information labels for body parts.

[0004] The sparse nature of millimeter-wave point clouds presents a challenge for feature extraction: limited by single-chip and antenna size, millimeter-wave point clouds are extremely sparse, making it difficult to perceive detailed human structure even with the naked eye. Extracting features containing human structural information (such as posture) using existing deep neural network models is challenging, directly impacting human anatomy tasks.

[0005] Specular reflection causes missing body parts in millimeter-wave point cloud data: Due to the small antenna aperture of low-cost millimeter-wave radar, most of the human body reflection signals do not return to the sensor, resulting in specular reflection, which leads to missing body parts in the point cloud and ultimately incorrect analysis results. Summary of the Invention

[0006] To address the aforementioned issues, this invention discloses a dynamic millimeter-wave radar point cloud human body parsing method based on joint learning. For millimeter-wave point cloud data, this invention first performs clustering, then uses a multi-task learning model to jointly perform human body parsing and pose estimation tasks for feature extraction, and then uses a non-local network to perform multi-task feature fusion. The final output is a point cloud with annotated semantic labels.

[0007] The specific technical solution is as follows:

[0008] Step 1: The sparsity of millimeter-wave point clouds is addressed through a multi-task feature extraction module; a multi-task learning model is used to jointly execute the main task of human parsing and the auxiliary task of pose estimation; the auxiliary task can effectively guide the human parsing network to extract high-level structural features representing the subject's pose; since there is a strong correlation between human pose and human parsing, human pose-related features help improve the accuracy and robustness of the parsing network in predicting semantic labels; for human parsing and pose estimation tasks, the multi-task learning model extracts the corresponding features in parallel; Step 2: The problem of missing body parts in point cloud data caused by specular reflection is addressed through a multi-task feature fusion module; inspired by nonlocal networks (NLN), a multi-task feature fusion method is designed, combining intra-task attention and inter-task attention mechanisms to aggregate the spatiotemporal features of the subject from a global perspective;

[0009] Step 3: Multi-task supervision. During the offline training phase of the system, this invention uses the Kinect system to obtain the true values ​​of human body parsing labels and pose estimation labels.

[0010] Furthermore, in step 1, the multi-task learning model can be divided into a point module, a frame module, and a feature aggregation module. The input of the model is a frame sequence of length s, each frame contains n points, and each point contains d feature dimensions.

[0011] Step 1.1: Human body analysis and feature extraction;

[0012] In the point module, for the point set C corresponding to the frame at time t... t Any radar point p in i,t The high-dimensional feature representation of points, i.e., point features, is obtained by using a multiple perceptron (MLP). The formula is:

[0013]

[0014] Where θ e represents the learnable parameters of the MLP, and H represents the human parsing task.

[0015] In the frame module, for each radar point p i,t Point features First, it is encoded into a higher-dimensional feature representation. The formula is:

[0016]

[0017] Where θ h Represents the learnable parameters of the MLP;

[0018] Step 1.2: Human posture feature extraction;

[0019] For human pose feature extraction, a slightly different network architecture is used compared to human parsing feature extraction; specifically, for the frame features of any frame... The process is performed using a Long Short-Term Memory (LSTM) network; the formula is as follows:

[0020]

[0021] Where θ r This represents the parameters of the LSTM.

[0022] Finally, Point features related to human pose tasks By connecting the features, we can obtain the feature vector of each point in the frame for the pose estimation task. The formula is:

[0023]

[0024] Furthermore, in step 1.1, the point features of all points in the frame are... Aggregated into frame features To extract global information from a frame; the formula is:

[0025]

[0026] Where N is the number of points in the frame corresponding to time t, and A() represents the attention function, which is θ a Represents the learnable parameters of the attention function;

[0027] Finally, the frame features Connect to point features Obtain the feature vector of each point in the frame under the human body analysis task. The formula is:

[0028]

[0029] Furthermore, step 2 uses two parallel NLNs to perform the human body parsing and pose estimation tasks, as detailed below:

[0030] Step 2.1: In-task attention mechanism. For the human body parsing task, the parsing NLN takes a series of parsing features as input and performs task self-attention to aggregate features from different frames. This generates a global context for classifying body parts in each frame to solve the problem of losing body parts due to mirror reflections in local frames.

[0031] Step 2.2: Inter-task attention mechanism; To fuse features from human parsing and pose estimation tasks, it is necessary to find the correlation between parsing features and pose features. An inter-task attention mechanism is adopted to calculate the spatiotemporal correlation between parsing features and pose features. For the parsing task, this method inputs the pose estimation features into the parsing NLN, and first processes the parsing feature matrix Z... H and pose feature matrix Z P A linear transformation is performed, followed by dot product and normalization of the result to obtain the inter-task attention matrix 'a' for the human body parsing task. H→P ;

[0032] Step 2.3 Feature Aggregation: Utilizing intra-task and inter-task attention matrices, this invention fuses human parsing features and pose estimation features from all frames to predict the body parts at a specific point in a frame.

[0033] Step 2.4: Model Output; Human Body Analysis Features Y H and pose estimation features Y P The data are processed by a multilayer perceptron and a fully connected neural network, respectively, and the final outputs are human body part classification information and human skeleton key point location information.

[0034] Furthermore, in step 2.1: For the parsing task, firstly, the point features of all points in the entire time series are... Stacked into feature matrix Z H After that, Z H The embedding vector is obtained through linear transformation. and Furthermore, to estimate the spatiotemporal correlation between points in each frame group, the embedding vectors are subjected to dot product and normalization using a nonlinear function to obtain the in-task attention matrix a for the human body parsing task. H The formula is as follows:

[0035]

[0036] Where σ represents a nonlinear function. and The parameters represent the linear transformation.

[0037] Similarly, in the pose estimation task, the same processing procedure yields the corresponding in-task attention matrix a. P The formula is as follows:

[0038]

[0039] Where σ represents a nonlinear function. and The parameter Z represents the linear transformation. PThis represents the point features of all points within all time series in the pose estimation task. The feature matrix obtained by stacking.

[0040] Furthermore, the specific formula for step 2.2 is as follows:

[0041]

[0042] in and The weight parameters represent the linear transformation.

[0043] Similarly, by inputting human parsing features into the pose estimation NLN, the inter-task attention matrix a for the pose estimation task is obtained. P→H The formula is as follows:

[0044]

[0045] in and The weight parameters represent the linear transformation.

[0046] Furthermore, in step 2.3, specifically for the human body analysis task, the feature matrix Z is first... H Linear transformations respectively are and Then, respectively with the in-task attention matrix a H Inter-task attention matrix a H→P Multiply the features to obtain intra-task features and inter-task features respectively, and calculate the weighted sum of all features based on their correlation with the current frame; finally, concatenate the intra-task features and inter-task features, and then combine the result with the original feature Z. H Element-by-element addition generates the final aggregated human body analytical feature Y. H The formula is as follows:

[0047]

[0048] in and These are the linear transformation parameters;

[0049] For the pose estimation task, the same method as described above is used to finally obtain the aggregated pose estimation features Y. P Its formula is:

[0050]

[0051] in and These are the parameters for the linear transformation.

[0052] Furthermore, it should be noted in step 3 that the Kinect system is only used in the offline training phase, but not in the inference phase; for the human body parsing task, cross-entropy loss is used to minimize the error between the predicted body part classification and the true classification at each point; the formula is as follows:

[0053]

[0054] Where N represents the number of points, and K is the number of semantic tag classes. It is a function that outputs either 0 or 1. A value of 1 indicates that sample n belongs to category k. It is the predicted probability that sample b belongs to category k;

[0055] For the pose estimation task, mean squared error is used to minimize the error between the predicted and actual positions of the skeletal joints; the formula is as follows:

[0056]

[0057] Where ||·|| represents L2 normal form, and p m Let represent the predicted and actual values ​​of the skeletal joints, respectively, and M represent the number of selected skeletal joints. The network architecture is trained end-to-end, and the overall supervision function of the system is:

[0058] L=γL H +βL P

[0059] Here, γ and β are hyperparameters.

[0060] The beneficial effects of this invention are:

[0061] This invention designs a dynamic millimeter-wave radar point cloud human body parsing method based on joint learning, which can generate point clouds with annotated semantic labels. It solves the problem that current human-centered millimeter-wave perception cannot obtain semantic information of millimeter-wave point clouds, achieving an accuracy of about 92% and an IoU accuracy of 84%. The predicted semantic labels can improve the performance of two downstream tasks (pose estimation and action recognition) by about 18% and 6%, respectively. Attached Figure Description

[0062] Figure 1 : Schematic diagram of the overall system structure.

[0063] Figure 2 : Schematic diagram of the multi-task feature extraction module structure.

[0064] Figure 3 : Schematic diagram of the multi-task feature fusion module structure.

[0065] Figure 4 System accuracy in different scenarios. Detailed Implementation

[0066] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0067] like Figure 1 As shown in the figure, this embodiment presents a dynamic millimeter-wave radar point cloud human body parsing method based on joint learning. The specific steps are as follows:

[0068] like Figure 2 As shown, step 1: Solve the sparsity of millimeter-wave point clouds through the "multi-task feature extraction module".

[0069] To address the issue of existing deep neural network models struggling to extract features containing human structural information due to the sparsity of millimeter-wave point clouds, this invention employs a multi-task learning model to jointly perform human parsing (the primary task) and pose estimation (an auxiliary task). The auxiliary task effectively guides the human parsing network to extract high-level structural features representing the subject's posture. Since there is a strong correlation between human pose and human parsing, pose-related features help improve the accuracy and robustness of the parsing network in predicting semantic labels. For both human parsing and pose estimation tasks, the multi-task learning model extracts the corresponding features in parallel. Specifically, the multi-task learning model can be divided into a point module, a frame module, and a feature aggregation module. The model's input is a frame sequence of length s, where each frame contains n points, and each point contains d feature dimensions.

[0070] Step 1.1: Human body analysis and feature extraction;

[0071] In the point module, for the point set C corresponding to the frame at time t... t Any radar point p in i,t The high-dimensional feature representation of points, i.e., point features, is obtained by using a multiple perceptron (MLP). The formula is:

[0072]

[0073] Where θ e represents the learnable parameters of the MLP, and H represents the human parsing task.

[0074] In the frame module, for each radar point p i,t Point features First, it is encoded into a higher-dimensional feature representation. The formula is:

[0075]

[0076] Where θ h This represents the learnable parameters of the MLP.

[0077] Furthermore, the point features of all points in the frame are... Aggregated into frame features To extract global information from a frame. The formula is:

[0078]

[0079] Where N is the number of points in the frame corresponding to time t, and A() represents the attention function, which is θ a This represents the learnable parameters of the attention function.

[0080] Finally, the frame features Connect to point features Obtain the feature vector of each point in the frame under the human body analysis task. The formula is:

[0081]

[0082] Step 1.2: Human posture feature extraction;

[0083] For human pose feature extraction, a slightly different network architecture is used compared to human parsing feature extraction. Specifically, for the frame features of any given frame... Processing is done using a Long Short-Term Memory (LSTM) network. The formula is:

[0084]

[0085] Where θ r This represents the parameters of the LSTM.

[0086] Finally, Point features related to human pose tasks By connecting the features, we can obtain the feature vector of each point in the frame for the pose estimation task. The formula is:

[0087]

[0088] like Figure 3 As shown, step 2: The "multi-task feature fusion module" is used to solve the problem of missing body parts in point cloud data caused by specular reflection.

[0089] To address the issue of missing body parts in point cloud data caused by specular reflection, this invention draws inspiration from Nonlocal Networks (NLNs) and designs a multi-task feature fusion method. This method combines intra-task attention and inter-task attention mechanisms to aggregate the spatiotemporal features of the subject from a global perspective. Specifically, this method uses two parallel NLNs to perform human body parsing and pose estimation tasks respectively.

[0090] Step 2.1: In-task attention mechanism;

[0091] For the human body parsing task, the parsing NLN takes a series of parsing features as input and performs task self-attention to aggregate features from different frames. This generates a global context for classifying body parts in each frame, addressing the problem of missing body parts due to specular reflections in local frames. More specifically, for the parsing task, it first processes point features from all points across all time series. Stacked into feature matrix Z H After that, Z H The embedding vector is obtained through linear transformation. and Furthermore, to estimate the spatiotemporal correlation between points in each frame group, the embedding vectors are subjected to dot product and normalization using a nonlinear function (such as the softmax function) to obtain the in-task attention matrix a for the human parsing task. H The formula is as follows:

[0092]

[0093] Where σ represents a nonlinear function. and The parameters represent the linear transformation.

[0094] Similarly, in the pose estimation task, the same processing procedure yields the corresponding in-task attention matrix a. P The formula is as follows:

[0095]

[0096] Where σ represents a nonlinear function. and The parameter Z represents the linear transformation. P This represents the point features of all points within all time series in the pose estimation task. The feature matrix obtained by stacking.

[0097] Step 2.2: Inter-task attention mechanism;

[0098] To integrate features from human parsing and pose estimation tasks, it is necessary to identify the correlation between parsing and pose features. This method employs an inter-task attention mechanism to calculate the spatiotemporal correlation between parsing and pose features. In implementation, for the parsing task, this method inputs the pose estimation features into a parsing NLN, first processing the parsing feature matrix Z... H and pose feature matrix Z P A linear transformation is performed, followed by dot product and normalization of the result to obtain the inter-task attention matrix 'a' for the human body parsing task. H→P The formula is as follows:

[0099]

[0100] in and The weight parameters represent the linear transformation.

[0101] Similarly, by inputting human parsing features into the pose estimation NLN, the inter-task attention matrix a for the pose estimation task is obtained. P→H The formula is as follows:

[0102]

[0103] in and The weight parameters represent the linear transformation.

[0104] Step 2.3: Feature aggregation;

[0105] By utilizing intra-task and inter-task attention matrices, this invention fuses human parsing features and pose estimation features from all frames to predict the body parts at a specific point in a frame.

[0106] Specifically, for the human body parsing task, the feature matrix Z is first... H Linear transformations respectively are and Then, respectively with the in-task attention matrix a H Inter-task attention matrix a H→P Multiply these features to obtain intra-task and inter-task features, respectively, and calculate a weighted sum of all features based on their correlation with the current frame. Finally, concatenate the intra-task and inter-task features, and sum the results with the original feature Z. H Element-by-element addition generates the final aggregated human body analytical feature Y. H The formula is as follows:

[0107]

[0108] in and These are the parameters for the linear transformation.

[0109] For the pose estimation task, the same method as described above is used to finally obtain the aggregated pose estimation features Y. P The formula is:

[0110]

[0111] in and These are the parameters for the linear transformation.

[0112] Step 2.4: Model Output;

[0113] Human body analysis features Y H and pose estimation features Y P The data are processed by a multilayer perceptron (MLP) and a fully connected neural network (FC), respectively, and the final outputs are human body part classification information and human skeleton key point location information.

[0114] Step 3: Multi-task supervision

[0115] During the offline training phase, this invention employs the Kinect system to obtain the true values ​​of the human body parsing labels and pose estimation labels. It should be noted that the Kinect system is only used during the offline training phase and is not needed during the inference phase. For the human body parsing task, this invention uses cross-entropy loss to minimize the error between the predicted body part classification and the true classification at each point. The formula is as follows:

[0116]

[0117] Where N represents the number of points, and K is the number of semantic tag classes. It is a function that outputs either 0 or 1. A value of 1 indicates that sample n belongs to category k. It is the predicted probability that sample n belongs to class k.

[0118] For attitude estimation tasks, this invention employs mean squared error (MSE) to minimize the error between the predicted and actual positions of the skeletal joints. The formula is as follows:

[0119]

[0120] Where ||·|| represents L2 normal form, and p m These represent the predicted and actual values ​​of the skeletal joints, respectively, and M represents the number of skeletal joints selected.

[0121] The network architecture designed in this invention is trained end-to-end, and the overall supervision function of the system is:

[0122] L=γL H +βL P

[0123] Here, γ and β are hyperparameters.

[0124] like Figure 4 The figure shows the system accuracy in different scenarios.

[0125] Example 1: Emergency Rescue

[0126] In emergency rescue operations, rescue team members often need to work closely together to accurately hand over tools to their teammates. However, in scenarios such as fire rescues, where there is a large amount of smoke, traditional camera-based imaging equipment is difficult to operate normally, while millimeter-wave imaging equipment remains robust in harsh environments. This invention improves the accuracy of radar in identifying a person's hands by adding additional semantic tags with body parts as extra input, thereby assisting rescue personnel in accurately completing tool handovers in smoke-filled environments.

[0127] Example 2: Action Recognition

[0128] In certain scenarios, precise perception of human movements is crucial. For example, in nursing homes, millimeter-wave devices used for health monitoring need to accurately identify falls in elderly residents. This invention improves the motion recognition performance of millimeter-wave devices by adding semantic information about body parts from point clouds.

[0129] Example 3: Identity Recognition

[0130] Due to privacy concerns associated with cameras, millimeter-wave devices have been increasingly deployed in private settings such as warehouses and offices in recent years to replace cameras for surveillance. Unlike camera imaging, which contains rich semantic information, millimeter-wave imaging is inherently sparsity, putting it at a disadvantage in identifying people. This invention improves the personnel identification performance of millimeter-wave devices by adding semantic information about body parts from point clouds.

[0131] Example 4: Autonomous Driving

[0132] In the field of autonomous driving, helping vehicles understand and recognize pedestrian movements is of great significance. This invention can be applied to vehicle-mounted millimeter-wave radar equipment to improve the ability of autonomous vehicles to recognize pedestrian movements and make timely predictions and responses when encountering emergencies.

[0133] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A dynamic millimeter-wave radar point cloud human body analysis method based on joint learning, characterized in that: Includes the following steps: Step 1: Solve the sparsity problem of millimeter-wave point clouds through a multi-task feature extraction module; A multi-task learning model is employed to jointly perform the main task of human parsing and the auxiliary task of pose estimation. The auxiliary task can effectively guide the human parsing network to extract high-level structural features representing the subject's pose. Since there is a strong correlation between human pose and human parsing, pose-related features help improve the accuracy and robustness of the parsing network in predicting semantic labels. For both human parsing and pose estimation tasks, the multi-task learning model extracts the corresponding features in parallel. Specifically, step 1 involves a multi-task learning model comprising a point module, a frame module, and a feature aggregation module. The input to the model is a sequence of lengths... A sequence of frames, each frame containing There are points, each containing _ points. Each feature dimension; Step 1.1: Human body analysis and feature extraction; In the point module, for The set of points corresponding to a time frame any radar point in A high-dimensional feature representation of points, i.e., point features, is obtained using a multi-presentation perceptron. The formula is: ； in This represents the learnable parameters of an MLP. This represents a human body analysis task; In the frame module, for each radar point Point features First, it is encoded into a higher-dimensional feature representation. The formula is: ； in Represents the learnable parameters of the MLP; Step 1.2: Human posture feature extraction; For human pose feature extraction, a slightly different network architecture is used compared to human parsing feature extraction; specifically, for the frame features of any frame... It uses a long short-term memory network for processing; The formula is: ； in Indicates the parameters of the LSTM; Finally, Point features related to human pose tasks By connecting the features, we can obtain the feature vector of each point in the frame for the pose estimation task. Its formula is: ； Step 2: Solve the problem of missing body parts in point cloud data caused by mirror reflection through a multi-task feature fusion module; Inspired by nonlocal networks, a multi-task feature fusion method is designed, which combines intra-task attention and inter-task attention mechanisms to aggregate the spatiotemporal features of the subject from a global perspective; Step 2 uses two parallel NLNs to perform human body parsing and pose estimation tasks respectively. Step 3: Multi-task supervision. During the offline training phase of the system, the Kinect system is used to obtain the true values ​​of human body parsing labels and pose estimation labels.

2. The method for dynamic millimeter-wave radar point cloud human body analysis based on joint learning according to claim 1, characterized in that: In step 1.1, the point features of all points in the frame are further... Aggregated into frame features To extract global information from the frame; The formula is: ;in for The number of points contained in the frame corresponding to a given time. Represents the attention function, which is Represents the learnable parameters of the attention function; Finally, the frame features Connect to point features To obtain the feature vector of each point in the frame under the human body analysis task. ; The formula is: 。 3. The method for dynamic millimeter-wave radar point cloud human body analysis based on joint learning according to claim 1, characterized in that: Step 2 is described in detail below: Step 2.1: In-task attention mechanism. For the human body parsing task, the parsing NLN takes a series of parsing features as input and performs task self-attention to aggregate features from different frames. This generates a global context for classifying body parts in each frame to solve the problem of losing body parts due to mirror reflections in local frames. Step 2.2: Inter-task attention mechanism; To fuse features from human parsing and pose estimation tasks, it is necessary to find the correlation between parsing features and pose features. An inter-task attention mechanism is adopted to calculate the spatiotemporal correlation between parsing features and pose features. For the parsing task, this method inputs the pose estimation features into the parsing NLN, and first processes the parsing feature matrix... and pose feature matrix A linear transformation is performed, followed by dot product and normalization of the result to obtain the inter-task attention matrix for the human parsing task. ; Step 2.3 Feature aggregation: Utilize intra-task and inter-task attention matrices to fuse human parsing features and pose estimation features from all frames to predict the body parts at a specific point in a frame; Step 2.4: Model Output; Human Body Feature Analysis and pose estimation features The data are processed by a multilayer perceptron and a fully connected neural network, respectively, and the final outputs are human body part classification information and human skeleton key point location information.

4. The method for dynamic millimeter-wave radar point cloud human body analysis based on joint learning according to claim 3, characterized in that: Step 2.1: For the parsing task, first, the point features of all points in the entire time series are... Stacked into feature matrices After that, The embedding vector is obtained through linear transformation. and Furthermore, to estimate the spatiotemporal correlation between points in each frame group, the embedding vectors are subjected to dot product and normalization using a nonlinear function to obtain the in-task attention matrix for the human body parsing task. The formula is as follows: ； in Represents a nonlinear function. and Parameters representing linear transformations; Similarly, in the pose estimation task, the same processing procedure yields the corresponding in-task attention matrix. The formula is as follows: ； in Represents a nonlinear function. and The parameters representing the linear transformation. This represents the point features of all points within all time series in the pose estimation task. The feature matrix obtained by stacking.

5. The method for dynamic millimeter-wave radar point cloud human body analysis based on joint learning according to claim 3, characterized in that: The specific formula for step 2.2 is as follows: ； in and The weighting parameters represent the linear transformation; Similarly, by inputting human parsing features into the pose estimation NLN, the inter-task attention matrix for the pose estimation task is obtained. The formula is as follows: ； in and The weight parameters represent the linear transformation.

6. The method for dynamic millimeter-wave radar point cloud human body analysis based on joint learning according to claim 3, characterized in that: Specifically, step 2.3, for the human body parsing task, firstly, the feature matrix... Linear transformations respectively are as follows: and Then, respectively with the in-task attention matrix Inter-task attention matrix Multiply the features to obtain intra-task features and inter-task features respectively, and calculate a weighted sum of all features based on their correlation with the current frame; finally, concatenate the intra-task features and inter-task features, and sum the results with the original features. Element-by-element addition generates the final aggregated human body analytical features. The formula is as follows: ； in and These are the linear transformation parameters; For the pose estimation task, the same method as described above is used to obtain the aggregated pose estimation features. ; The formula is: ； in and These are the parameters for the linear transformation.

7. The method for dynamic millimeter-wave radar point cloud human body analysis based on joint learning according to claim 1, characterized in that: Step 3 requires clarification: the Kinect system is only used for the offline training phase, and is not needed during the inference phase; for the human body parsing task, cross-entropy loss is used to minimize the error between the predicted body part classification and the true classification for each point; the formula is as follows: ； in Indicates the number of points. The number of semantic tag classes. It is a function that outputs either 0 or 1. A value of 1 indicates that the sample Category , It is a sample Category The predicted probability; For the pose estimation task, mean squared error is used to minimize the error between the predicted and actual positions of the skeletal joints; the formula is as follows: ； in Indicates L2 normal form, and Let represent the predicted and actual values ​​of the skeletal joints, respectively, and M represent the number of selected skeletal joints. The network architecture is trained end-to-end, and the overall supervision function of the system is: ； in, and It's a hyperparameter.

Images