Method and apparatus for behavior recognition based on millimeter wave radar point cloud features
By clustering and reconstructing key points from millimeter-wave radar point cloud data, and combining this with a neural network trained on depth camera labels, the problem of environmental interference in millimeter-wave radar action recognition was solved, achieving highly accurate and visualized action recognition.
Patent Information
- Application Number
- CN202410400875.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-03
- Publication Date
- 2026-02-27
- Estimated Expiration
- 2044-04-03
AI Technical Summary
Existing millimeter-wave radar is susceptible to interference from static environments in motion recognition, leading to abnormal motion recognition and a lack of visualization effects.
By collecting millimeter-wave radar point cloud data, DBSCAN clustering was used to segment the data into individual data. The PointNet+AM+CNN network was trained with Kinect DK depth camera labels to reconstruct human key points, and TCN temporal network was used for behavior recognition.
It improves the accuracy of action recognition and enables the visualization of action recognition.
Smart Images
Figure CN118135663B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to a millimeter wave radar point cloud feature extraction and classification recognition method and device. BACKGROUND
[0002] Millimeter wave radar is a radar working in a millimeter wave band (1-10 mm) for detection, and has been developed since the 1940s, and is one of important means in the field of modern wireless sensing. Compared with cameras, WiFi and other means, the millimeter wave radar has the characteristics of high precision and strong anti-interference. In recent years, the technology in the field of radar has developed rapidly, from basic ranging and speed measurement to environmental perception. The millimeter wave radar has been widely used in the fields of automobiles, consumers, industries and medical treatment.
[0003] Action recognition is also a relatively advanced research in the field of millimeter wave radar. At present, the action classification recognition is mainly performed by analyzing the frequency spectrum extracted from the original radar data. However, the interference in different environments will also be different, which will affect the extracted frequency spectrum and cause the action recognition to be abnormal. The application processes the point cloud data generated by the millimeter wave radar, can effectively weaken the static environment interference, and can form a visual effect for action recognition. SUMMARY
[0004] The application aims to overcome the above-mentioned defects of the prior art, and provides a behavior recognition method and device based on millimeter wave radar point cloud features.
[0005] The application solves the technical problems by adopting the following technical solutions:
[0006] A first aspect of the application relates to a behavior recognition method based on millimeter wave radar point cloud features, comprising the following steps:
[0007] Step 1: collecting millimeter wave radar point cloud original data, and dividing the original data into multiple single-person data by clustering for subsequent tasks, and simultaneously using a Kinect DK depth camera to collect real data as a model training label;
[0008] Step 2: forming a human key point reconstruction network by improving a PointNet neural network, reconstructing each frame of point cloud data to form a human key point model;
[0009] Step 3: collecting the continuous frame human key point data generated in step 2, constructing a TCN (time sequence convolution network), and inputting the key point data to realize behavior recognition.
[0010] Step 1 specifically comprises:
[0011] 1) Deploy the millimeter wave radar device in a good view location (e.g. 2-2.5m high, facing the area to be measured, while the elevation angle is 15-30°), configure the related parameters of millimeter wave radar FMCW according to the needs, and collect the point cloud data (position data based on radar coordinate system) when multiple people move.
[0012] 2) Perform coordinate system conversion task on the original point cloud data:
[0013]
[0014] Where (x i ,y i ,z i ) represents the original point cloud data collected by the radar, Δx, Δy, Δz respectively represent the corresponding coordinate axis position difference of the radar device relative to the origin of the real coordinate system. (x′ i ,y′ i ,z′ i ) represents the point cloud position data after the coordinate conversion task.
[0015] 3) For the scene of multiple people moving, use DBSCAN clustering algorithm to perform clustering operation on the processed point cloud data:
[0016] DBSCAN(D,Eps,MinPts) (2)
[0017] The preset radius Eps and minimum density threshold MinPts are used to perform clustering operation on the point cloud data in the multiple person scene (effect as Figure 1 ), and the point cloud clustering data, i.e. single person data, is obtained.
[0018] 4) While deploying the radar device, use Microsoft Kinect DK depth camera to collect real key point data [(GT_x1,GT_y1,GT_z1),(GT_x2,GT_y2,GT_z3),…,(GT_x i ,GT_y i ,GT_z i ),…,(GT_x 25 ,GT_y 25 ,GT_z 25 )] under human motion state, which respectively represent Figure 2 the real reference value of the corresponding 25 human key point coordinates, which are used as the labels for model training in step 2.
[0019] Step 2 specifically includes:
[0020] 1) Get the point cloud data after clustering from each frame in step 1, calculate the centroid position of each cluster data, and record it as (c_x i ,c_y i ,c_zi )。
[0021] 2) Construct a neural network, add CNN and attention mechanism (PointNet+AM+CNN) based on the classic point cloud classification segmentation network PointNet. The CNN network has 4 layers, the first two layers are two-dimensional convolution layers with channel numbers of 16 and 32 respectively, and the last two layers are two fully connected layers containing 512 neurons.
[0022] 3) Train the network model. Collect the data after clustering in step 1, combine with the real data collected by Kinect DK depth camera, form the key point reconstruction data set. The key point reconstruction model finally outputs a prediction sequence of length 25 [(p_x1, p_y1, p_z1), (p_x2, p_y2, p_z3), …, (p_x i ,p_y i ,p_z i ),…,(p_x 25 ,p_y 25 ,p_z 25 )]. Where (p_x i ,p_y i ,p_z i ) represents the coordinate prediction value of the i-th key point in the human body in Figure 2
[0023] Step 3 includes:
[0024] 1) First set the cluster detection threshold d th , combine the calculated centroid position in step 2 step 1, calculate the centroid distance between adjacent frame data clusters, realize the continuous tracking of each cluster data, that is, extract the continuous motion data of each target.
[0025]
[0026] The above formula is used to calculate the distance between clusters in adjacent two frames. If it is less than the detection threshold d th , it is determined as the same cluster data, that is, the data of the same person.
[0027] 2) Construct a time series convolutional neural network TCN, the main body is an 8-layer residual structure, each residual structure is sequentially executed by two dilated convolution modules with a convolution kernel of 5*5, and the dilated factor is 2 i , where i is the current residual structure layer number. In each residual structure, there is also a 1-dimensional convolution layer with a convolution kernel of 3*3, which performs time constraint and strengthens the connection between the front and rear data. After the residual structure, there are three fully connected layers to realize the final classification task.
[0028] 3) Train the TCN model, a total of 100 iterations, the basic learning rate is 0.001, the learning rate is reduced by 10 times every 5 times, save the model once at the end of each iteration, and select the model with the highest classification accuracy as the classification model of the application. The model finally outputs the action recognition result.
[0029] The second aspect of the application relates to a behavior recognition device based on millimeter wave radar point cloud features, characterized by comprising a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the behavior recognition method based on millimeter wave radar point cloud features of the application.
[0030] The third aspect of the application relates to a computer readable storage medium having a program stored thereon, which is executed by a processor to implement the behavior recognition device based on millimeter wave radar point cloud features of the application.
[0031] The improved key point reconstruction network and TCN time series network in the application have good effects, improve the accuracy of action recognition based on point cloud data, and make the action recognition task visualized.
[0032] The application has the following beneficial effects:
[0033] (1) Based on the millimeter wave radar point cloud data, the human key point reconstruction makes the action recognition task visualized.
[0034] (2) The improved key point reconstruction network and TCN time series network have good effects, and improve the accuracy of action recognition based on point cloud data. BRIEF DESCRIPTION OF DRAWINGS
[0035] Figure 1 is the effect diagram of the point cloud data after clustering in the method of the application, for each cluster data, different colors are used for distinction, and a rectangular frame is used for marking
[0036] Figure 2 is a reference diagram of the human key point model reconstructed based on point cloud data in the application
[0037] Figure 3 is the overall flowchart of the application. DETAILED DESCRIPTION
[0038] The technical solutions of the application will be further described below with reference to the drawings.
[0039] Example 1
[0040] This embodiment relates to a behavior recognition method based on millimeter wave radar point cloud features, comprising the following steps:
[0041] Step 1: Collect millimeter wave radar point cloud raw data, and divide the raw data into multiple single person data through clustering for subsequent tasks, and use the Kinect DK depth camera to collect real data as model training labels;
[0042] Step 2: Form a human key point reconstruction network by improving the PointNet neural network, reconstruct each frame of point cloud data to form a human key point model;
[0043] Step 3: Collect continuous frame human key point data generated in step 2, construct a TCN (temporal convolutional network), and input the key point data to realize behavior recognition.
[0044] Step 1 specifically includes:
[0045] 1) Deploy the millimeter wave radar device in a good view position (for example, 2-2.5m high, facing the area to be measured, and the elevation angle is 15-30°), configure the related parameters of the millimeter wave radar FMCW according to the requirements, and collect the point cloud data (position data based on the radar coordinate system) when multiple people move.
[0046] 2) Perform coordinate system conversion task on the point cloud raw data:
[0047]
[0048] Where (x i ,y i ,z i ) represents the original point cloud data collected by the radar, Δx, Δy, Δz respectively represents the position difference of the corresponding coordinate axis of the radar device relative to the origin of the real coordinate system. (x′ i ,′ i ,z′ i ) represents the point cloud position data after the coordinate conversion task.
[0049] 3) For the scene of multiple people moving, use the DBSCAN clustering algorithm to perform clustering operation on the processed point cloud data:
[0050] DBSCAN(D,Eps,MinPts) (2)
[0051] The preset radius Eps and the minimum density threshold MinPts are used to cluster the point cloud data in the multiple person scene (the effect is as Figure 1 ), and the point cloud clustering data, i.e. single person data, is obtained.
[0052] 4) During the model training phase, while deploying radar equipment, it is necessary to use the Microsoft Kinect DK depth camera to collect real keypoint data of human motion states [(GT_x1,GT_y1,GT_z1),(GT_x2,GT_y2,GT_z3),…,(GT_x...]. i ,GT_y i ,GT_z i ),…,(GT_x 25 ,GT_y 25 ,GT_z 25 )], respectively represent Figure 1 The corresponding 25 human body key point coordinates are used as real reference values and are used as labels for model training in step 2.
[0053] Step 2 specifically includes:
[0054] 1) Obtain the clustered point cloud data for each frame from step 1, calculate the centroid position of each cluster, and record it as (c_x i ,c_y i ,c_z i ).
[0055] 2) Construct a neural network by adding a CNN and attention mechanism to the classic point cloud classification and segmentation network PointNet (PointNet+AM+CNN). The CNN network has 4 layers. The first two layers are two-dimensional convolutional layers with 16 and 32 channels, respectively, and the last two layers are two fully connected layers with 512 neurons each.
[0056] 3) Train the network model. Collect the clustered data from each frame in step 1, and combine it with the real data captured by the Kinect DK depth camera to form a keypoint reconstruction dataset. The keypoint reconstruction model finally outputs a prediction sequence of length 25 [(p_y1,p_y1,p_z1),(p_x2,p_y2,p_z3),…,(p_x... i ,p_y i ,p_z i ),…,(p_x 25 ,p_y 25 ,p_z 25 (p_x) i ,p_y i ,p_z i )express Figure 2 Predicted coordinates of the i-th key point in the human body
[0057] Step 3 specifically includes:
[0058] 1) First, set the cluster detection threshold d. th, combined with the centroid position calculated in step 2, step 1, the centroid distance between adjacent frame data clusters is calculated, and the continuous tracking of each cluster data is realized, that is, the continuous motion data of each target is extracted.
[0059]
[0060] The above formula is used to calculate the distance between clusters between two adjacent frames, which is less than the detection threshold d th Then it is determined as the same cluster data, that is, the data of the same person.
[0061] 2) Construct a time convolution neural network (TCN), the main body of which is an 8-layer residual structure, each residual structure of which is sequentially executed by two dilated convolution modules with a convolution kernel of 5*5 and an expansion factor of 2 i Where i is the current residual structure layer. In each residual structure, there is also a 1D convolution layer with a convolution kernel of 3*3, which performs time constraint and strengthens the connection between the front and rear data. After the residual structure, there are three fully connected layers for realizing the final classification task.
[0062] 3) Train the TCN model, a total of 100 iterations, the basic learning rate is 0.001, the learning rate is reduced by 10 times every 5 times, the model is saved once at the end of each iteration, and the model with the highest classification accuracy is selected as the classification model of the application. The model finally outputs the action recognition result.
[0063] Embodiment 2
[0064] The embodiment relates to a behavior recognition device based on millimeter wave radar point cloud features, characterized by comprising a memory and one or more processors, the memory has executable code stored therein, and the one or more processors execute the executable code to implement the behavior recognition method based on millimeter wave radar point cloud features.
[0065] Embodiment 3
[0066] The embodiment relates to a computer-readable storage medium, which has a program stored thereon, and the program is executed by a processor to implement the behavior recognition method based on millimeter wave radar point cloud features.
[0067] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.
Claims
1. A method for behavior recognition based on millimeter wave radar point cloud features, comprising the following steps: Step 1: Collecting millimeter wave radar point cloud raw data, and dividing the raw data into multiple single-person data by DBSCAN clustering for subsequent tasks, while using a Kinect DK depth camera to collect real data as model training labels; Step 2: Forming a human key point reconstruction network by improving the PointNet neural network to reconstruct each frame of point cloud data and form a human key point model; specifically including: 21) From the clustered point cloud data of each frame obtained in step 1, the centroid position of each cluster data is calculated and recorded as ( ); 22) Building a neural network, adding CNN and attention mechanism to the classic point cloud classification and segmentation network PointNet, namely PointNet+AM+CNN; the CNN network has 4 layers, the first two layers are two-dimensional convolution layers with channel numbers of 16 and 32 respectively, and the last two layers are two fully connected layers with 512 neurons; 23) Train the network model; collect the data after clustering each frame in step 1, combine the real data collected by the Kinect DK depth camera to form the key point reconstruction data set; the key point reconstruction model finally outputs a predicted sequence with a length of 25 [ ), ( ), …, ( ), …, ( )];Wherein( ) represents the coordinate prediction value of the i-th key point of the human body; Step 3: Collecting continuous frame human key point data generated in step 2, building a time series convolution network TCN, and inputting the key point data to realize behavior recognition; specifically including: 31) First set the cluster detection threshold , combined with the calculated centroid position in step 21), the centroid distance between adjacent frame data clusters is calculated to realize continuous tracking of each cluster data, that is, to extract the continuous motion data of each target; （3） The above formula is used to calculate the inter-cluster distance between two adjacent frames, which is less than the detection threshold Then it is determined as the same cluster data, i.e. the data of the same person. 32) Construct a timing convolutional neural network (TCN), the main body is an 8-layer residual structure, each residual structure is sequentially executed by two dilated convolution modules with a convolution kernel of 5*5, and the dilated factor is where i is the current residual structure layer number; In each residual structure, there is also a 1-dimensional convolution layer with a convolution kernel of 3*3, which performs time constraint and strengthens the connection between the data before and after; After the residual structure, there are three fully connected layers to realize the final classification task; 33) Training the TCN model for a total of 100 iterations, with a basic learning rate of 0.001, a learning rate reduction of 10 times every 5 times, and a model saved once at the end of each iteration, and the model with the highest classification accuracy is selected as the classification model; the model finally outputs the classification result of the action. 2.The method of claim 1, wherein, Step 1 specifically includes: 11) Deploying the millimeter wave radar device in a location with good visibility, configuring the related parameters of the millimeter wave radar FMCW according to the requirements, and collecting point cloud data during multi-person movement; 12) Performing coordinate system conversion task on the point cloud raw data: （1） wherein (x, y, z) represents the original point cloud data collected by the radar, ) represents the original point cloud data collected by the radar, respectively represent the corresponding coordinate axis position differences of the radar device relative to the coordinate origin of the real coordinate system; ) represents the point cloud position data after the coordinate conversion task. 13) For multi-person activity scenes, using DBSCAN clustering algorithm to perform clustering operation on the processed point cloud data: （2） Setting the radius Eps and the minimum density threshold MinPts to perform clustering operation on the point cloud data in the multi-person scene to obtain point cloud clustering data, i.e. single-person data; 14) The real key point data of human motion state is collected by Microsoft Kinect DK depth camera while deploying radar equipment represent the real reference values of 25 human key point coordinates, respectively, as the labels for model training in step 2. 3. The behavior recognition apparatus based on millimeter wave radar point cloud features, characterized in that, A memory and one or more processors, the memory having stored therein executable code, the one or more processors executing the executable code to implement the method for behavior recognition based on millimeter wave radar point cloud features of any one of claims 1-2.
4. A computer-readable storage medium, characterized in that, A program is stored thereon, which is executed by a processor to implement the method for behavior recognition based on millimeter wave radar point cloud features of any one of claims 1-2.