Millimeter wave radar fine-grained human body posture perception method based on multi-dimensional feature extraction

Abstract

The application discloses a millimeter wave radar fine-grained human body posture perception method based on multi-dimensional feature extraction and belongs to the field of computer vision and radar perception, and comprises the following steps: generating 4D point cloud by a millimeter wave radar sensor through a device access module; performing multi-target tracking on the 4D point cloud by a target tracking module, predicting the target position by using a Kalman filter, dividing the point cloud attribution by a gate function, creating a new target by adopting DBSCAN clustering and eliminating the target continuously not attributed to the point cloud, and obtaining the target center position; and converting the point cloud to a local coordinate system based on the target center position and the 4D point cloud, normalizing the point cloud, fusing continuous multi-frame point clouds, and inputting the deep learning network based on the improved PointNet and U-Net of the attention mechanism to output the human body posture joint node coordinates. The application improves the human body posture detection precision under the limited data mode and realizes the human body health monitoring function available in a family environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and radar perception, and particularly relates to a millimeter wave radar fine-grained human body pose perception method based on multi-dimensional feature extraction. BACKGROUND

[0002] Precise perception and accurate algorithms are the key to realizing automated human health monitoring, and the rapid development of sensor technology and computer performance makes it possible. In the field of intelligent monitoring, computer vision technology based on deep learning has a relatively mature solution, and the human body pose recognition and reconstruction solution based on computer vision also has high accuracy. However, the computer vision-based solution is greatly affected by occlusion and has a certain degree of privacy risk, and people may not be able to universally accept the deployment of visual sensors in the bedroom, bathroom and other areas, so as to not be able to completely cover the home environment. Therefore, to truly realize all-weather human health monitoring in the home environment, we still need to find a better solution.

[0003] Millimeter wave radar provides a new way to solve these problems. First, millimeter wave radar uses frequency-modulated continuous wave to continuously monitor the target, and it is not affected by visible light and has strong anti-interference ability. Second, unlike the visual solution, millimeter wave radar does not directly capture complete target information and has no risk of leaking user privacy. Finally, millimeter wave radar itself is relatively low in cost, and the family can easily afford the cost of millimeter wave radar, so millimeter wave radar is a suitable human health monitoring sensor for the family. However, millimeter wave radar relies on a small number of antennas to transmit and receive signals, and the amount of environmental information it can obtain is less than that of visual sensors. Currently, there is a lack of deep learning perception methods for limited data modalities, and the output result accuracy is poor, which cannot be applied to actual scenarios. SUMMARY

[0004] To solve the above technical problems, the application provides a millimeter wave radar fine-grained human body pose perception method based on multi-dimensional feature extraction, which improves the human body pose detection accuracy under limited data modalities, realizes the function of human health monitoring available in the home environment, and has no risk of privacy leakage.

[0005] To achieve the above purpose, the application provides a millimeter wave radar fine-grained human body pose perception method based on multi-dimensional feature extraction, which includes:

[0006] S1, managing a millimeter wave radar sensor to generate a 4D point cloud;

[0007] S2, performing multi-target tracking on the 4D point cloud to obtain a target center position, wherein the multi-target tracking performs the following sub-steps:

[0008] S21, predicting the center position of the tracking target by using a Kalman filter;

[0009] S22, dividing the point cloud to the tracking target by a gating function based on the predicted center position and updating the target center and velocity;

[0010] S23, creating a new tracking target by using DBSCAN clustering on the unassigned point cloud and based on the number of points and a signal-to-noise ratio threshold;

[0011] S24, removing the tracking target that has not been assigned with the point cloud for a continuous time;

[0012] S3, performing human pose detection based on the target center position and the 4D point cloud to output human pose joints, wherein the human pose detection performs the following sub-steps:

[0013] S31, converting the point cloud to a local coordinate system with the target center as the origin and normalizing;

[0014] S32, fusing the point cloud of continuous T frames;

[0015] S33, inputting the fused point cloud into an improved PointNet and U-Net network to regress joint coordinates.

[0016] Optionally, the process of managing the millimeter wave radar sensor comprises:

[0017] Through the control interface, the antenna array and the signal processing link are interacted with the millimeter wave radar hardware, including sensor online state checking, transmission and reception parameter and frame rate configuration, distance-velocity-angle imaging parameter setting, signal processing algorithm parameter adjustment and point cloud data receiving, and clock alignment and external parameter calibration of multiple radar devices.

[0018] Optionally, in S21, the process of predicting the center position of the tracking target by using the Kalman filter comprises:

[0019] A state vector is constructed using the target position and velocity, the target position at the current time is predicted based on a state transition model, and the target position is updated by a measurement model, wherein the state transition model is based on the assumption of uniform acceleration, and the measurement model is based on the observation value obtained by the millimeter wave radar data.

[0020] Optionally, in S22, the process of dividing the point cloud by the gating function comprises:

[0021] A three-dimensional ellipsoid region is set as a relevant region for each tracking target, and the ellipsoid radius is dynamically adjusted based on the predicted center position and the prediction error; for the disputed points located in the overlapping region of multiple ellipsoids, the score of the point to the target center is calculated by a scoring mechanism, and the point cloud is assigned to the target with the highest score; and the target center and velocity are updated based on the centroid of the assigned point cloud.

[0022] Optionally, in S23, the process of creating a new tracking target by DBSCAN clustering includes:

[0023] DBSCAN clustering is performed on the unattributed point cloud to generate a cluster region, the sum of the point cloud quantity and the signal-to-noise ratio of each cluster region is calculated, and when the sum of the point cloud quantity and the signal-to-noise ratio reaches a preset threshold, a new tracking target is created at the center of the cluster region.

[0024] Optionally, in S24, the process of removing a tracking target that has not been attributed to a point cloud for a continuous time includes:

[0025] The number of frames for which each tracking target has not been attributed to a point cloud is monitored, and when the number of frames reaches a preset value, the target is removed from the set of tracking targets.

[0026] Optionally, in S31, the process of converting the point cloud to a local coordinate system and normalizing includes: obtaining the target center position from the target tracking module, translating the point cloud to a local space with the target center as the origin, and performing scale normalization and outlier removal.

[0027] Optionally, in S33, the process of the improved PointNet and U-Net network includes:

[0028] The point cloud embedding feature extraction module improves PointNet based on the self-attention mechanism and residual connection to extract multi-scale features of the point cloud; the human body pose feature extraction module is based on the U-Net encoder-decoder structure, and fuses multi-scale features through downsampling and upsampling and skip connection; the human body pose output module maps the features to joint coordinates through MLP.

[0029] The present application discloses a millimeter wave radar fine-grained human body pose perception method based on multi-dimensional feature extraction, generates 4D point cloud through a millimeter wave radar sensor, adopts a multi-target tracking method combining Kalman filtering and a gate function, effectively overcomes the problems of point cloud sparsity and noise interference, realizes stable human body target tracking and positioning, through local coordinate normalization and multi-frame point cloud time sequence fusion, combined with an improved PointNet and U-Net deep learning network, the accuracy and robustness of human body joint detection are significantly improved, the system has strong anti-interference ability, is not affected by light conditions and shielding, at the same time avoids the privacy leakage risk existing in visual sensors, provides a reliable all-weather human body health monitoring solution for a family environment. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application illustrated in the drawings, and their description, are presented to explain the application and are not intended to limit the application. In the drawings:

[0031] Figure 1 A flowchart of a millimeter wave radar fine-grained human body pose perception method based on multi-dimensional feature extraction according to an embodiment of the present application is shown in FIG. 1.

[0032] Figure 2 A dynamic human body target tracking algorithm based on a Kalman filter according to an embodiment of the present application is shown in FIG. 2.

[0033] Figure 3 A neural network module structure of a human body pose joint detection algorithm based on deep learning according to an embodiment of the present application is shown in FIG. 3.

[0034] Figure 4 A tool structure for constructing various modules according to an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0035] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0036] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0037] As shown in FIG. 1, a millimeter wave radar fine-grained human body pose perception method based on multi-dimensional feature extraction is provided in the present embodiment, which includes: Figure 1

[0038] S1, managing a millimeter wave radar sensor to generate a 4D point cloud;

[0039] S2, performing multi-target tracking on the 4D point cloud to obtain a target center position, wherein the multi-target tracking performs the following sub-steps:

[0040] S21, predicting the center position of a tracked target using a Kalman filter;

[0041] S22, dividing the point cloud to the tracked target by a gating function based on the predicted center position and updating the target center and velocity;

[0042] S23, adopting DBSCAN clustering for unattributed point clouds and creating a new tracked target based on a point quantity and a signal-to-noise ratio threshold;

[0043] S24, eliminating a tracked target that has not been attributed to a point cloud for a continuous time;

[0044] S3, performing human body pose detection based on the target center position and the 4D point cloud to output a human body pose joint, wherein the human body pose detection performs the following sub-steps: ​

[0045] S31, convert the point cloud to a local coordinate system with the target center as the origin and normalize it;

[0046] S32, fuse the continuous T-frame point clouds;

[0047] S33, input the fused point cloud into the improved PointNet and U-Net network to regress the joint coordinates.

[0048] Further, the process of managing the millimeter wave radar sensor includes:

[0049] Through the control interface, interact with the millimeter wave radar hardware, configure the antenna array and signal processing link, including sensor online state viewing, transceiver parameter and frame rate configuration, distance-speed-angle imaging parameter setting, signal processing algorithm parameter adjustment and point cloud data reception, and clock alignment and external parameter calibration of multiple radar devices.

[0050] Further, in S21, the process of predicting the target center position using a Kalman filter includes:

[0051] Using the target position and velocity to construct a state vector, predicting the target position at the current time based on a state transition model, and updating the target position through a measurement model, wherein the state transition model is based on the assumption of uniform acceleration, and the measurement model is based on the observation value obtained from the millimeter wave radar data.

[0052] Further, in S22, the process of dividing the point cloud by a gate function includes:

[0053] Draw a three-dimensional ellipsoid region as a relevant region for each tracking target, dynamically adjust the ellipsoid radius based on the predicted center position and the prediction error; for the disputed points located in the overlapping region of multiple ellipsoids, calculate the score of the point to the target center through a scoring mechanism, and assign the point cloud to the target with the highest score; update the target center and velocity based on the centroid of the attributed point cloud.

[0054] Further, in S23, the process of creating a new tracking target using DBSCAN clustering includes:

[0055] DBSCAN clustering of unattributed point clouds generates a cluster region, calculates the sum of the number of point clouds and the signal-to-noise ratio of each cluster region, and when the sum of the number of point clouds and the signal-to-noise ratio reaches a preset threshold, creates a new tracking target with the cluster region center.

[0056] Further, in S24, the process of removing tracking targets that have not been attributed to point clouds continuously includes:

[0057] Monitor the number of frames for which each tracking target has not been attributed to point clouds, and when the number of frames reaches a preset value, remove the target from the set of tracking targets.

[0058] Further, in S31, the process of converting the point cloud to the local coordinate system and normalizing includes: obtaining the target center position from the target tracking module, translating the point cloud to the local space with the target center as the origin, and performing scale normalization and outlier rejection.

[0059] Further, in S33, the process of the improved PointNet and U-Net network includes:

[0060] The point cloud embedding feature extraction module improves PointNet based on the self-attention mechanism and residual connection to extract multi-scale features of the point cloud; the human body pose feature extraction module is based on the U-Net encoder-decoder structure, and fuses multi-scale features through downsampling and upsampling and skip connection; the human body pose output module maps the features to joint coordinates through MLP.

[0061] Specifically, the implementation process of the embodiment includes:

[0062] As shown in Figure 1 The application embodiment provides a millimeter wave radar fine-grained human body pose perception method and tool based on multi-dimensional feature extraction, which includes:

[0063] The device access module is used for unified management, configuration and data access of the millimeter wave radar sensor, and guarantees the stable input and timing consistency of the subsequent algorithm. Through device initialization and control instructions, the module can assist in controlling the electromagnetic wave transmission characteristics of the millimeter wave radar and various parameters of the signal processing algorithm, and generate point cloud data according to environmental data acquisition and basic preprocessing.

[0064] Further, the device access module is responsible for communication with the millimeter wave radar sensor, receiving sensor data, and transmitting data to other modules for subsequent processing. In this module, we define various sensor control instructions, and the command format is composed of command type and sensor configuration parameters. The device access module judges the millimeter wave radar device model according to the electrical characteristics of the device and converts the control instructions to the control instructions matched with the hardware device. The command type is divided into setting command and control command, which has different functions and purposes in millimeter wave radar hardware control. The setting command is used to control the electromagnetic wave transmission characteristics of the millimeter wave radar and various parameters of the signal processing algorithm, such as the start and stop frequency of the frequency-modulated continuous wave, the bandwidth, the frequency sweeping mode, the frame rate, the transmission power, the antenna array enabling strategy, and the threshold parameters of noise statistics adaptive adjustment such as constant false alarm detection, SNR and RCS calculation. The control command is used to control the functions of the millimeter wave radar hardware device, such as enabling and suspending the millimeter wave radar, device detection, temperature and power supply state query, etc., which is used to detect whether the sensor is working normally and obtain the current state information of the sensor.

[0065] Specifically, in the above embodiment, the target tracking module is used to dynamically detect and track human body targets in the millimeter wave radar field of view, output the current tracking human body target coordinate position, and according to the point cloud data segmented by the human body target, the single execution process of the algorithm is as shown in Figure 2

[0066] S1, input the point cloud and the center position of the current tracking target set , and use the Kalman filter to predict the center position of each tracking target .

[0067] S2, taking the target center position predicted in S1 and the point cloud data as the input of point cloud attribution prediction, performing point cloud attribution through a gating function , dividing the point cloud to the corresponding tracking target, and updating the center and speed of the target based on the centroid of the target attribution point cloud.

[0068] S3, taking the point cloud data divided in S2 as the input, using DBSCAN to cluster the remaining point cloud that is not attributed, and calculating the center of the cluster that meets the point quantity threshold and the signal-to-noise ratio SNR threshold as a candidate target to create a new tracking target.

[0069] S4, taking the tracking targets in S1 and S2 and the divided point cloud data as the input, removing the target that does not obtain point cloud attribution for a continuous preset number of frames from the tracking target set.

[0070] The target dynamic tracking algorithm mainly includes three stages, namely predicting the target position, dividing the target point cloud, and creating a candidate target. In the stage of predicting the target position, the algorithm applies the Kalman filter prediction algorithm to each tracking target. When applying the Kalman filter, the algorithm uses the position and speed of the target to construct the covariance matrix of the Kalman filter, and predicts the position of each dynamic target at the current time. The state vector of the Kalman filter is defined as a six-dimensional vector of the target position, speed and acceleration:

[0071] ;

[0072] wherein represents the target position, represents the target speed. According to the current target position provided by the dynamic target tracking algorithm , assuming that the tracking target moves at a uniform acceleration within a continuous time, the Kalman filter target state transition model can be defined as follows:

[0073] ​​ ;

[0074] State transition matrix is defined as follows:

[0075] ;

[0076] wherein, is the time step, which is usually determined according to the execution frequency of the target tracking algorithm. is the process noise, the sampling of the process noise follows wherein, is the process noise covariance matrix, which is defined as follows:

[0077] ;

[0078] wherein, is the variance of the acceleration noise.

[0079] The Kalman filter measurement model updates the tracking target position based on the target state transition model as follows:

[0080] ;

[0081] wherein, the observation matrix is defined as follows:

[0082] ;

[0083] wherein is the measurement noise, the measurement noise follows wherein, is the noise covariance matrix, which is obtained by the device access module through the millimeter wave radar data.

[0084] In the target point cloud partitioning stage, the embodiment first uses a gate function to demarcate a relevant area for each tracking target, and considers that the point cloud in the relevant area may belong to the tracking target. The gate function demarcates a three-dimensional ellipsoid area for each tracking target:

[0085] ;

[0086] wherein, is the target center position predicted by the Kalman filter , is the radius of the ellipsoid in three directions, which is dynamically adjusted based on the uncertainty of the target, and the dynamic adjustment calculation method is as follows:

[0087] ;

[0088] wherein, respectively are Kalman filter prediction error covariance matrices corresponding diagonal elements, is a scale factor used to control the size of the gate.

[0089] For the point cloud of non-controversial regions, the embodiment directly divides the point cloud into the tracking target. For the point cloud of the overlapping part of the gate function of different tracking targets, the embodiment uses a scoring mechanism to evaluate which tracking target each point belongs to. The scoring mechanism is designed as follows:

[0090] ;

[0091] wherein, is the distance from the point to the center of the target, is the distance normalization factor. The distance normalization factor is calculated by the prediction error of the Kalman filter , and the calculation method is as follows:

[0092] ;

[0093] By dynamically matching the distance normalization factor with the prediction error of the Kalman filter, the gate function can be dynamically matched with the uncertainty of the target, and thus the point cloud matching algorithm can be dynamically adjusted and adapted according to different targets and scenes.

[0094] Calculate the score of each controversial point to each target After that, the controversial point is assigned to the target with the highest score. For the point cloud that does not belong to any gate function, the embodiment does not assign it to any target, and these point clouds will enter the candidate target creation stage.

[0095] In the candidate target creation stage, the embodiment uses the DBSCAN clustering method to cluster the point cloud without a target, and then calculates the number of point clouds and the sum of SNRs of each clustering region. If the sum of the number of point clouds and the sum of SNRs of the DBSCAN clustering region reaches the target threshold, a new tracking target is created in this clustering region.

[0096] Specifically, in the above embodiment, the human pose detection module is the core module of the tool of the embodiment, and outputs the human pose joint coordinate result based on the point cloud. The human pose detection module is composed of point cloud data preprocessing and a deep learning neural network. The point cloud preprocessing part performs normalization operation on the target belonging point cloud according to the target dynamic tracking human center position, and fuses multiple frame point cloud data. The deep learning neural network part uses the preprocessed point cloud data as input, and outputs the final result of the human pose.

[0097] ​​​Furthermore, in the point cloud data preprocessing section, this embodiment first performs point cloud normalization on a single frame. For the tracking target's... Frame point cloud The algorithm first obtains the center position of the current target from the target tracking module. Then, for the point cloud to which the current target belongs. Translate to The local space is the origin of the coordinate system. For the multi-frame point cloud fusion part, this embodiment uses the fusion... arrive continuous Frame point cloud. Before fusion, the algorithm first... arrive Based on the current target center location Normalize, then use all The point clouds in the frames are connected together and used as a data frame input into the neural network.

[0098] The human pose detection neural network receives preprocessed point cloud data and outputs the final result of human pose. The core algorithm structure diagram of this module is shown below. Figure 3 As shown in the figure, this model extracts local features from point clouds based on an improved PointNet and handles the order independence of point cloud data. It also introduces an improved U-Net network to learn global and local features of the point cloud, resulting in a more complex network structure, a larger number of parameters, and stronger feature extraction capabilities. The network structure mainly consists of three parts: a point cloud embedding feature extraction module, a human pose feature extraction module, and a human pose output module.

[0099] Specifically, in the above embodiments, the point cloud embedding feature extraction module is implemented based on an improved PointNet architecture to extract multi-scale features while maintaining the order independence of point cloud data. The module includes an attention sublayer, residual connection units, a multilayer perceptron (MLP) sublayer, and a global pooling unit. To replace the STNkd spatial transformation network module in the original PointNet, this embodiment introduces a self-attention mechanism to characterize the correlation between points. Specifically, a query is generated for the input point set. ,key AND value and adopt The self-attention calculation mode of the attention sublayer is improved. Since the attention calculation keeps the permutation invariance of the input points, the sublayer can model the relationship between points without relying on the point sequence, thereby reducing the influence of point cloud disorder on feature learning. Meanwhile, in order to enhance the training stability of the deep network and alleviate the gradient vanishing, a residual connection is provided between the output of the attention sublayer and its input, forming a short-circuit path of identity or linear projection. This structure improves the feature propagation efficiency and optimization convergence without destroying the permutation invariance. After the attention sublayer and the residual connection unit, the module further extracts higher-level abstract features through a multi-layer perceptron. The MLP is composed of multiple fully connected layers and nonlinear activation functions, which is used to learn the nonlinear mapping relationship between point cloud features and to perform feature reorganization, dimension expansion or compression on the attention output. Finally, in order to obtain a global representation that is permutation invariant to the point set, the module uses max pooling to aggregate the point-level features and outputs the point cloud embedding features. The global feature can be used as the input of the pose feature extraction module.

[0100] Specifically, in the above embodiment, the human pose feature extraction module is implemented based on an improved U-Net architecture, which is used to jointly model the correlation between point cloud features and human poses in global and local perspectives, and to interact and fuse information between high-dimensional and low-dimensional feature spaces. The module includes a high-dimensional feature extraction unit and a multi-dimensional fusion feature extraction unit, and is realized through an encoder-decoder symmetric structure and a cross-layer skip connection mechanism for end-to-end training. In this embodiment, the high-dimensional feature extraction unit is implemented in the U-Net encoder path, which is composed of multiple levels of downsampling blocks. Each level includes a multi-layer perceptron or a residual module, as well as a nonlinear activation and normalization layer, and the input is the point-level feature output by the point cloud embedding feature extraction module. After level-by-level downsampling and channel expansion, the encoder can extract high-dimensional abstract semantic features with gradually increasing scales while retaining the intermediate features of each level as skip connection feature caches for feature backfilling in the decoder path. In the U-Net decoder path, the multi-dimensional fusion feature extraction unit is implemented, which is composed of multiple levels of upsampling blocks and forms a symmetric structure with the encoder path in depth. Each upsampling block includes an interpolation or feature propagation operator, a skip connection splicing with the corresponding encoder layer, a fusion through an MLP or a residual module layer, and a nonlinear activation and normalization layer. This unit gradually upsamples the high-dimensional semantic features to higher spatial resolution and multi-scale fuses them with low-dimensional detailed features, taking into account the global skeleton relationship and local key point neighborhood geometry details of the human pose. The skip connection provides edge, topology and local structure information, and the fused features are used to restore the pose-related fine-grained information. Multi-scale receptive fields are introduced in each level of fusion to adaptively adjust the contribution from global features and local features, so as to realize dynamic selection of multi-dimensional features.

[0101] Specifically, in the above embodiment, the human pose output module is configured to transform the fused features into human pose related representation information, and the human pose output result is composed of key points. The human pose output module learns the per-keypoint channel features using an MLP and converts the feature vectors of the human pose feature extraction module into final human pose key points.

[0102] Specifically, in the above embodiment, the tool constructed based on the above modules is shown in Figure 4 :

[0103] The tool summary design is divided into five levels, from top to bottom, which are user interaction layer, service interface layer, algorithm access layer, basic software layer and device access layer. The user interaction layer is the interface for the user to interact with the tool, including the graphical operation interface and the user interaction interface based on Web. Through the user interaction layer, the user can access and control the functions of the tool, such as configuring tool parameters, viewing tool status, viewing tool processing results, etc. The service interface layer is used to provide services for the user interface layer, while hiding the specific implementation details of the functions of the tool. The services of the service interface layer provide access security for the tool, and abstract independent services for the user to view system data, control device work, etc. The algorithm access layer is the core functional module of the tool, which includes the human dynamic recognition and tracking algorithm module and the human pose recognition algorithm module. When the millimeter wave radar point cloud is transmitted to the algorithm access layer, the human target dynamic tracking algorithm updates the target tracking state according to the human dynamic target cache, and the human pose detection algorithm generates the corresponding human pose, and writes the algorithm result into the persistent storage through the persistent storage service. The basic software layer provides basic software support for the functions of the tool, and the functions and services of the tool are realized based on the basic software layer. The device access layer is used to control the access of the millimeter wave radar device, which shields the specific details of the millimeter wave radar hardware to the upper layer, while allowing the upper layer to control the millimeter wave radar function and monitor the millimeter wave radar working state. Through such a tool architecture, the tool can realize a user-friendly interaction interface, efficiently realize human target tracking and pose detection functions, ensure the safety of the tool, and provide basic function support.

[0104] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for millimeter wave radar fine-grained human pose perception based on multi-dimensional feature extraction, characterized in that, Comprising: S1, managing a millimeter wave radar sensor to generate a 4D point cloud; S2, performing multi-target tracking on the 4D point cloud to obtain a target center position, wherein the multi-target tracking performs the following sub-steps: S21, predicting the center position of the tracked target using a Kalman filter; S22, dividing the point cloud into a tracked target by a gating function based on the predicted center position and updating the target center and speed; S23, using DBSCAN clustering on unattributed point clouds and creating new tracking targets based on point quantity and signal-to-noise ratio thresholds; S24, eliminating tracking targets that have not been continuously attributed to point clouds; S3, performing human pose detection based on the target center position and the 4D point cloud to output human pose joint nodes, wherein the human pose detection performs the following sub-steps: S31, converting the point cloud to a local coordinate system with the target center as the origin and normalizing; In S31, the process of converting the point cloud to the local coordinate system and normalizing includes: obtaining the target center position from the target tracking module, translating the point cloud to the local space with the target center as the origin, and performing scale normalization and outlier elimination; S32, fusing consecutive T-frame point clouds; S33, inputting the fused point cloud into an improved PointNet and U-Net network to regress joint node coordinates; In S33, the process of the improved PointNet and U-Net network includes: The point cloud embedding feature extraction module improves PointNet based on self-attention mechanism and residual connection to extract multi-scale point cloud features; the human pose feature extraction module is based on the U-Net encoder-decoder structure, and fuses multi-scale features through downsampling and upsampling and skip connection; the human pose output module maps the features to joint node coordinates through MLP.

2. The millimeter wave radar fine-grained human pose perception method based on multi-dimensional feature extraction according to claim 1, wherein The process of managing the millimeter wave radar sensor includes: Interacting with the millimeter wave radar hardware through a control interface, configuring the antenna array and signal processing link, including sensor online state viewing, transceiver parameter and frame rate configuration, distance-velocity-angle imaging parameter setting, signal processing algorithm parameter adjustment and point cloud data reception, and clock alignment and external parameter calibration for multiple radar devices.

3. The millimeter wave radar fine-grained human pose perception method based on multi-dimensional feature extraction according to claim 1, wherein In S21, the process of predicting the center position of the tracked target using a Kalman filter includes: Using the target position and velocity to construct a state vector, predicting the target position at the current time based on a state transition model, and updating the target position through a measurement model, wherein the state transition model is based on the assumption of uniform acceleration, and the measurement model is based on the observation values obtained from the millimeter wave radar data.

4. The millimeter wave radar fine-grained human pose perception method based on multi-dimensional feature extraction according to claim 1, wherein In S22, the process of dividing the point cloud by a gating function includes: A three-dimensional ellipsoid region is drawn as a relevant region for each tracking target, and the ellipsoid radius is dynamically adjusted based on the predicted center position and the prediction error; for controversial points located in the overlapping region of multiple ellipsoids, the score of the point to the target center is calculated through a scoring mechanism, and the point cloud is assigned to the target with the highest score; the target center and velocity are updated based on the centroid of the attributed point cloud.

5. The millimeter wave radar fine-grained human body pose perception method based on multi-dimensional feature extraction according to claim 1, wherein, In S23, the process of creating a new tracking target by DBSCAN clustering includes: DBSCAN clustering is performed on the unattributed point cloud to generate a cluster region, the sum of the point cloud quantity and the signal-to-noise ratio of each cluster region is calculated, and when the sum reaches a preset threshold, a new tracking target is created at the center of the cluster region.

6. The millimeter wave radar fine-grained human body pose perception method based on multi-dimensional feature extraction according to claim 1, wherein, In S24, the process of removing tracking targets that have not been attributed to point clouds continuously includes: The number of frames in which each tracking target has not been attributed to point clouds continuously is monitored, and when the number of frames reaches a preset value, the target is removed from the set of tracking targets.

Images