Radar human activity recognition method, device and electronic equipment based on deep learning
By processing radar data based on deep learning, extracting three-dimensional features of multiple angles and performing human body movement recognition, the problem of accuracy and low efficiency of radar human body movement recognition in the prior art is solved, and more efficient human body movement recognition is achieved.
Patent Information
- Application Number
- CN202410883532.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-03
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-03
AI Technical Summary
The prior art only uses a single method to process radar data in radar human activity recognition, resulting in less effective feature extraction, high computational complexity, and low accuracy and efficiency of action recognition.
Using a deep learning-based method, millimeter-wave radar echo data from multiple angles is collected, and three-dimensional data (distance, time and Doppler speed) is obtained through processing, and a pre-trained deep learning model is used to recognize human movements, including features The extraction layer, the Transformer layer and the recognition layer perform spatial angle feature extraction and attention feature fusion to reduce the computational complexity.
Effective use of multi-dimensional data improves the accuracy and efficiency of human movement recognition, and can better distinguish complex movements from adapting to movement changes at different angles.
Smart Images

Figure CN118409313B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of human activity recognition, and in particular to a radar human activity recognition method, device and electronic equipment based on deep learning. Background Art
[0002] Radar-based human activity recognition (HAR) technology has attracted great attention from researchers in recent years. The core of this technology is to use radar signals to capture and analyze various human activities. At present, the existing technology only uses a single method to process the features of radar data. The effective features extracted from radar data are few, the calculation complexity is high, and the accuracy and efficiency of action recognition are low. Summary of the invention
[0003] The present invention provides a radar human activity recognition method, device and electronic device based on deep learning, so as to solve the defects that the prior art only adopts a single method to perform feature processing on radar data, extracts fewer effective features from the radar data, has high calculation complexity, and has low accuracy and efficiency in action recognition.
[0004] In a first aspect, the present invention provides a radar human activity recognition method based on deep learning, comprising:
[0005] Collect millimeter-wave radar echo data from multiple angles reflected by the human body;
[0006] Processing the millimeter-wave radar echo data at the multiple angles to obtain three-dimensional data at the multiple angles, wherein the three-dimensional data includes data in three dimensions: distance, time, and Doppler velocity;
[0007] Inputting the three-dimensional data at multiple angles into a pre-built deep learning model to obtain human motion recognition results corresponding to the three-dimensional data at multiple angles output by the deep learning model;
[0008] Among them, the deep learning model is trained based on sample three-dimensional data from multiple angles and human motion recognition result labels of the sample three-dimensional data from multiple angles.
[0009] In some embodiments, the deep learning model includes a feature extraction layer, a Transformer layer, and a recognition layer;
[0010] Correspondingly, the inputting the three-dimensional data of the multiple angles into a pre-built deep learning model to obtain the human action recognition results corresponding to the three-dimensional data of the multiple angles output by the deep learning model includes:
[0011] Inputting the three-dimensional data of each angle into the feature extraction layer, and obtaining the spatial angle features corresponding to the three-dimensional data of each angle output by the feature extraction layer;
[0012] Encoding the spatial angle feature corresponding to the three-dimensional data of each angle to obtain the spatial angle feature vector corresponding to the three-dimensional data of each angle;
[0013] Inputting the spatial angle feature vectors corresponding to the three-dimensional data at multiple angles into the Transformer layer, and obtaining the attention features corresponding to the three-dimensional data at multiple angles output by the Transformer layer;
[0014] The attention feature is input into the recognition layer to obtain the human motion recognition results corresponding to the three-dimensional data of the multiple angles output by the recognition layer.
[0015] In some embodiments, the feature extraction layer includes a projection layer, a two-dimensional feature extraction layer, and a feature fusion layer;
[0016] Correspondingly, the inputting of the three-dimensional data of each angle into the feature extraction layer to obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature extraction layer includes:
[0017] Input the three-dimensional data of each angle into the projection layer, and obtain a distance-speed two-dimensional graph, a time-distance two-dimensional graph, and a time-speed two-dimensional graph corresponding to the three-dimensional data of each angle output by the projection layer;
[0018] Inputting the distance-speed two-dimensional graph, the time-distance two-dimensional graph and the time-speed two-dimensional graph corresponding to the three-dimensional data of each angle into the two-dimensional feature extraction layer, and obtaining the distance-speed feature graph, the time-distance feature graph and the time-speed feature graph corresponding to the three-dimensional data of each angle output by the two-dimensional feature extraction layer;
[0019] The distance-speed feature graph, time-distance feature graph and time-speed feature graph corresponding to the three-dimensional data of each angle are input into the feature fusion layer to obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature fusion layer.
[0020] In some embodiments, the distance-speed feature graph, the time-distance feature graph, and the time-speed feature graph corresponding to the three-dimensional data of each angle are input into the feature fusion layer to obtain the spatial angle feature corresponding to the three-dimensional data of each angle output by the feature fusion layer, including:
[0021] Compressing the distance-speed characteristic graph, the time-distance characteristic graph and the time-speed characteristic graph corresponding to the three-dimensional data of each angle to obtain corresponding distance-speed characteristic compression graph, time-distance characteristic compression graph and time-speed characteristic compression graph;
[0022] Using an attention mechanism, the attention weights of the distance-speed feature compression map, the time-distance feature compression map, and the time-speed feature compression map are obtained respectively;
[0023] Based on the distance-speed feature map, time-distance feature map and time-speed feature map corresponding to the three-dimensional data of each angle, and the attention weights of the distance-speed feature compression map, time-distance feature compression map and time-speed feature compression map, feature dot product is performed to obtain the spatial angle features corresponding to the three-dimensional data of each angle.
[0024] In some embodiments, the recognition layer includes a multi-layer perceptron MLP layer and a softmax classification layer;
[0025] Correspondingly, the inputting the attention feature into the recognition layer to obtain the human action recognition results corresponding to the three-dimensional data of the multiple angles output by the recognition layer includes:
[0026] Performing feature compression on the attention feature to obtain a one-dimensional attention feature;
[0027] Using an MLP layer to perform nonlinear mapping on the one-dimensional attention feature to obtain a processed one-dimensional attention feature;
[0028] A softmax classification layer is used to perform action recognition on the processed one-dimensional attention features to obtain action categories corresponding to the three-dimensional data at multiple angles.
[0029] In some embodiments, collecting millimeter wave radar echo data at multiple angles reflected by a human body includes:
[0030] An antenna array consisting of multiple receiving antennas is used to collect millimeter-wave radar echo data at multiple angles reflected by the human body;
[0031] The processing of the millimeter wave radar echo data at the multiple angles to obtain three-dimensional data at the multiple angles includes:
[0032] Performing beamforming processing on the millimeter-wave radar echo data at the multiple angles to obtain three-dimensional radar data at the multiple angles;
[0033] Perform two-dimensional fast Fourier transform processing on the three-dimensional radar data at the multiple angles to obtain the three-dimensional data at the multiple angles.
[0034] In some embodiments, performing two-dimensional fast Fourier transform processing on the three-dimensional radar data at the multiple angles to obtain the three-dimensional data at the multiple angles includes:
[0035] Perform one-dimensional fast Fourier transform on the three-dimensional radar data of each angle along the distance dimension to obtain distance information;
[0036] Perform one-dimensional fast Fourier transform on the three-dimensional radar data at each angle along the Doppler velocity dimension to obtain velocity information at different distances;
[0037] Based on the distance information and the speed information, transformed two-dimensional data is obtained, and based on the transformed two-dimensional data, three-dimensional data at each angle is obtained.
[0038] In some embodiments, the process of determining the deep learning model includes:
[0039] Obtain sample millimeter wave radar echo data at multiple angles reflected by the human body;
[0040] Processing the sample millimeter-wave radar echo data at the multiple angles to obtain sample three-dimensional data at the multiple angles, wherein the sample three-dimensional data includes sample data in three dimensions: distance, time, and Doppler velocity;
[0041] Determine human action recognition result labels of the sample three-dimensional data of the multiple angles;
[0042] Using the sample three-dimensional data of the multiple angles as training samples and the human action recognition result labels of the sample three-dimensional data of the multiple angles as sample labels to train an initial deep learning model;
[0043] Iteratively optimize the parameters of the initial deep learning model to obtain the deep learning model.
[0044] In a second aspect, the present invention further provides a radar human activity recognition device based on deep learning, comprising:
[0045] A data acquisition unit, used to collect millimeter-wave radar echo data at multiple angles reflected by the human body;
[0046] A data processing unit, used to process the millimeter-wave radar echo data at multiple angles to obtain three-dimensional data at multiple angles, wherein the three-dimensional data includes data in three dimensions: distance, time, and Doppler velocity;
[0047] An action recognition unit, configured to input the three-dimensional data at multiple angles into a pre-built deep learning model, and obtain a human action recognition result corresponding to the three-dimensional data at multiple angles output by the deep learning model;
[0048] Among them, the deep learning model is trained based on sample three-dimensional data from multiple angles and human motion recognition result labels of the sample three-dimensional data from multiple angles.
[0049] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a radar human activity recognition method based on deep learning as described above is implemented.
[0050] The present invention provides a radar human activity recognition method, device and electronic device based on deep learning. The method collects millimeter-wave radar echo data at multiple angles reflected by the human body, processes the millimeter-wave radar echo data at multiple angles, and obtains three-dimensional data at multiple angles. The three-dimensional data includes data in three dimensions of distance, time and Doppler velocity. A pre-trained deep learning model is used to perform human motion recognition based on the three-dimensional data at multiple angles to obtain corresponding human motion recognition results. The method can effectively utilize multi-dimensional data, reduce the complexity of calculation, and improve the accuracy and efficiency of human motion recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0052] Figure 1 This is one of the flow charts of the radar human activity recognition method based on deep learning provided by the present invention;
[0053] Figure 2 is a schematic diagram of beamforming provided by an embodiment of the present invention;
[0054] Figure 3 is a schematic diagram of a two-dimensional fast Fourier transform provided by an embodiment of the present invention;
[0055] Figure 4 This is the second flow chart of the radar human activity recognition method based on deep learning provided by the present invention;
[0056] Figure 5 is a schematic diagram of projecting three-dimensional data at each angle along different dimensions provided by an embodiment of the present invention;
[0057] Figure 6 is a schematic diagram of a process of feature fusion based on an attention mechanism provided by an embodiment of the present invention;
[0058] Figure 7 is a flowchart of a process for determining a deep learning model provided by an embodiment of the present invention;
[0059] Figure 8 is a schematic diagram of the structure of a radar human activity recognition device based on deep learning provided by an embodiment of the present invention;
[0060] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0062] With the development of deep learning technology, the use of deep learning models to process millimeter wave radar data can automatically identify human activity patterns. However, this field faces a major challenge. Compared with conventional optical image data, the collection and annotation process of sample radar data is more complicated, and the cost of model training is high. Therefore, it is particularly important to develop a method that can learn and extract as many features as possible from a limited radar data set.
[0063] Common signals used for radar human activity recognition include 2D time-frequency spectrum, 2D distance-angle diagram and 3D time-distance-Doppler velocity diagram. Among them, neither the 2D time-frequency spectrum nor the 2D distance-angle diagram can completely contain all the information of the radar signal. Although the 3D time-distance-Doppler velocity cube includes more radar information, the increase in dimension makes the features represented by the 3D time-distance-Doppler velocity cube more sparse than those represented by the 2D time-frequency spectrum diagram, etc., so it is difficult to extract the key information represented by the 3D time-distance-Doppler velocity cube by directly applying the 3D convolutional neural network. In addition, the data processing amount of directly using the three-dimensional cube is much more than that of the 2D convolutional neural network, so it is difficult to achieve satisfactory results.
[0064] To this end, the embodiments of the present invention provide a radar human activity recognition method, device and electronic device based on deep learning. By collecting millimeter-wave radar echo data at multiple angles reflected by the human body, the millimeter-wave radar echo data at multiple angles are processed to obtain three-dimensional data at multiple angles, and the three-dimensional data includes data in three dimensions of distance, time and Doppler velocity. A pre-trained deep learning model is used to perform human motion recognition based on the projection of the three-dimensional data at multiple angles to obtain corresponding human motion recognition results. This method can effectively utilize multi-dimensional data, reduce the complexity of calculation, and improve the accuracy and efficiency of human motion recognition.
[0065] Figure 1 is one of the flow charts of the radar human activity recognition method based on deep learning provided by the present invention, such as Figure 1 As shown, the method includes the following steps 110, 120 and 130.
[0066] Step 110: Collect millimeter-wave radar echo data at multiple angles reflected by the human body.
[0067] It should be noted that millimeter-wave radar detects human movement by measuring the frequency changes of microwave signals reflected by the human body. Millimeter-wave radar has the characteristics of high precision, high resolution, high penetration and privacy protection. It can work under various environmental conditions and is less affected by interference such as bad weather and lighting environment.
[0068] Step 120: Process the millimeter-wave radar echo data at multiple angles to obtain three-dimensional data at multiple angles, where the three-dimensional data includes data in three dimensions: distance, time, and Doppler velocity.
[0069] Optionally, beamforming, two-dimensional fast Fourier transform and other processing are performed on the millimeter-wave radar echo data at multiple angles.
[0070] It can be understood that by processing the millimeter-wave radar echo data at multiple angles, data in three dimensions of distance, time and Doppler velocity can be obtained, which can provide more comprehensive and richer information for subsequent human motion recognition.
[0071] In some embodiments, collecting millimeter wave radar echo data at multiple angles reflected by a human body includes:
[0072] An antenna array consisting of multiple receiving antennas is used to collect millimeter-wave radar echo data at multiple angles reflected by the human body;
[0073] Process the millimeter-wave radar echo data at multiple angles to obtain three-dimensional data at multiple angles, including:
[0074] Perform beamforming processing on millimeter-wave radar echo data at multiple angles to obtain three-dimensional radar data at multiple angles;
[0075] The three-dimensional radar data at multiple angles are processed by two-dimensional fast Fourier transform to obtain three-dimensional data at multiple angles.
[0076] Optionally, receiving antennas may be distributed at different locations to obtain millimeter-wave radar echo data at different angles.
[0077] Beamforming refers to synthesizing the signals received by multiple receiving antennas to form the desired ideal signal. By adjusting the phase and amplitude of the signal of each antenna element, a main beam can be formed in a specific direction, thereby achieving directional transmission and reception of signals, which can improve the directionality and strength of the signal.
[0078] Figure 2 A schematic diagram of beamforming provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the four receiving antennas are at a certain distance Arrange them to form an antenna array, and the receiving incident angle is The incident wave signal.
[0079] Optionally, the expression of the beamformed signal is as follows:
[0080]
[0081]
[0082] in, is the weight matrix corresponding to the received signal of each receiving antenna, is the received signal of each receiving antenna before beamforming, is the angle of incidence, is the wavelength of the incident wave, is the distance between the receiving antennas, is an imaginary unit, is the signal after beamforming.
[0083] Optionally, the number of receiving antennas is 4, and the receiving signals of the 4-channel receiving antennas are combined into a single channel through beamforming.
[0084] Figure 3 A schematic diagram of a two-dimensional fast Fourier transform provided in an embodiment of the present invention is shown in FIG. Figure 3 As shown, in some embodiments, two-dimensional fast Fourier transform processing is performed on three-dimensional radar data at multiple angles to obtain three-dimensional data at multiple angles, including:
[0085] Perform one-dimensional fast Fourier transform on the three-dimensional radar data of each angle along the distance dimension to obtain distance information;
[0086] Perform one-dimensional fast Fourier transform on the three-dimensional radar data at each angle along the Doppler velocity dimension to obtain velocity information at different distances;
[0087] Based on the distance information and the speed information, transformed two-dimensional data is obtained, and based on the transformed two-dimensional data, three-dimensional data at each angle is obtained.
[0088] Among them, the horizontal axis of the transformed two-dimensional data represents the radial distance from the human body to the radar, and the vertical axis represents the speed of the human body relative to the radar; if the energy of the coordinate (r, v) in a grid in the two-dimensional matrix is significant, it means that the human body reflection at the distance r from the radar is obvious, and the human body moves at an instantaneous speed v relative to the radar.
[0089] It should be noted that, considering the movement of the human body in front of the radar, different parts of the human body are at different distances from the radar, and the transformed two-dimensional data contains rich human body information in the distance dimension; at the same time, different parts of the limbs have different speeds and movement directions, and the transformed two-dimensional data contains identification clues in the speed dimension; considering the distance and speed dimensions comprehensively, the transformed two-dimensional data contains rich human body movement information.
[0090] Step 130: input the three-dimensional data at multiple angles into a pre-built deep learning model to obtain human motion recognition results corresponding to the three-dimensional data at multiple angles output by the deep learning model.
[0091] Among them, the deep learning model is trained based on sample three-dimensional data from multiple angles and human motion recognition result labels of sample three-dimensional data from multiple angles.
[0092] It should be noted that human motion recognition based on three-dimensional data from multiple angles and deep learning models can help distinguish complex movements, especially those that exhibit similar features at different angles, and improve the accuracy of human motion recognition.
[0093] It should be noted that the deep learning model trained based on sample data from multiple angles and the corresponding human motion recognition result labels can better adapt to the changes in motion at different angles and improve the accuracy and robustness of recognition.
[0094] In an embodiment of the present invention, millimeter-wave radar echo data at multiple angles reflected by the human body are collected and processed to obtain three-dimensional data at multiple angles, wherein the three-dimensional data includes data in three dimensions: distance, time, and Doppler velocity. A pre-trained deep learning model is used to perform human motion recognition based on the projection of the three-dimensional data at multiple angles to obtain corresponding human motion recognition results. This method can fully explore the information changes of human motion in the spatial orientation dimension, effectively utilize multi-dimensional data, reduce the complexity of calculation, and improve the accuracy and efficiency of human motion recognition.
[0095] Figure 4 The second flow chart of the radar human activity recognition method based on deep learning provided by the present invention is as follows: Figure 4 As shown, in some embodiments, the deep learning model includes a feature extraction layer, a Transformer layer, and a recognition layer.
[0096] Correspondingly, step 130 inputs the three-dimensional data of multiple angles into a pre-built deep learning model to obtain the human motion recognition results corresponding to the three-dimensional data of multiple angles output by the deep learning model, including the following steps 131, 132, 133 and 134.
[0097] Step 131: input the three-dimensional data of each angle into the feature extraction layer, and obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature extraction layer.
[0098] In some embodiments, the feature extraction layer includes a projection layer, a two-dimensional feature extraction layer, and a feature fusion layer.
[0099] Correspondingly, step 131 inputs the three-dimensional data of each angle into the feature extraction layer to obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature extraction layer, including the following steps 1311, 1312 and 1313.
[0100] Step 1311: input the three-dimensional data of each angle into the projection layer, and obtain the distance-speed two-dimensional graph, time-distance two-dimensional graph and time-speed two-dimensional graph corresponding to the three-dimensional data of each angle output by the projection layer.
[0101] Figure 5 A schematic diagram of projecting three-dimensional data of each angle along different dimensions provided by an embodiment of the present invention. Figure 5 As shown, the three dimensions of the three-dimensional data at each angle correspond to the time dimension, velocity (Doppler) dimension and distance dimension respectively. Each dimension contains a specific physical meaning. By performing average projection along the three directions, three two-dimensional heat maps with different semantics can be obtained.
[0102] Specifically, 1) the average projection is performed along the time dimension to obtain a two-dimensional distance-speed graph, which represents the average distance and speed probability distribution within the period of time; 2) the average projection is performed along the speed dimension to obtain a two-dimensional time-distance graph, which represents the change of human body distance over time; 3) the average projection is performed along the distance dimension to obtain a two-dimensional time-speed graph, which represents the change of human body movement speed over time.
[0103] It can be understood that by inputting three-dimensional data at each angle into the projection layer, three two-dimensional images with different semantics corresponding to the three-dimensional data at each angle output by the projection layer are obtained, which achieves data compression while maintaining key information. It not only effectively solves the problem of sparse effective information in three-dimensional space, but also significantly reduces the amount of network parameters and the complexity of data while fully mining the characteristics of multi-view three-dimensional data, thereby significantly reducing computational overhead.
[0104] Step 1312: input the distance-speed two-dimensional graph, time-distance two-dimensional graph and time-speed two-dimensional graph corresponding to the three-dimensional data of each angle into the two-dimensional feature extraction layer, and obtain the distance-speed feature graph, time-distance feature graph and time-speed feature graph corresponding to the three-dimensional data of each angle output by the two-dimensional feature extraction layer.
[0105] Optionally, a 2D CNN is used to build a neural network framework to obtain a two-dimensional feature extraction layer, which includes a convolution layer, a Relu activation function layer and a Maxpooling layer. The shallow and deep features are connected through a shortcut layer.
[0106] Step 1313: input the distance-speed feature graph, time-distance feature graph and time-speed feature graph corresponding to the three-dimensional data of each angle into the feature fusion layer to obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature fusion layer.
[0107] It can be understood that by integrating the features in different feature maps through the feature fusion layer, a more comprehensive and rich feature expression can be obtained, which helps to remove redundant information in different feature maps and retain the most critical features, thereby improving the accuracy of action recognition.
[0108] In an embodiment of the present invention, the time-distance-Doppler velocity three-dimensional data are projected onto three corresponding planes to obtain three two-dimensional images of different viewing angles, and three convolutional neural networks are used to extract key information of the three-dimensional images. The key information is transmitted to the fusion neural network, which ensures that the key information in the three-dimensional data is not lost and greatly reduces the computational load of the neural network. By integrating different two-dimensional images, the dynamic behavior of the target object can be more comprehensively understood, thereby improving the accuracy and reliability of radar human activity recognition.
[0109] Figure 6 The following is a flow chart of feature fusion based on the attention mechanism provided in an embodiment of the present invention. Figure 6 As shown, in some embodiments, the distance-speed feature graph, the time-distance feature graph, and the time-speed feature graph corresponding to the three-dimensional data of each angle are input into the feature fusion layer to obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature fusion layer, including:
[0110] Compressing the distance-speed characteristic graph, the time-distance characteristic graph and the time-speed characteristic graph corresponding to the three-dimensional data of each angle to obtain the corresponding distance-speed characteristic compression graph, the time-distance characteristic compression graph and the time-speed characteristic compression graph;
[0111] The attention mechanism is used to obtain the attention weights of the distance-speed feature compression map, the time-distance feature compression map, and the time-speed feature compression map respectively;
[0112] Based on the distance-speed feature map, time-distance feature map and time-speed feature map corresponding to the three-dimensional data of each angle, as well as the attention weights of the distance-speed feature compression map, time-distance feature compression map and time-speed feature compression map, the feature dot product is performed to obtain the spatial angle feature corresponding to the three-dimensional data of each angle (i.e. Figure 6 attention features).
[0113] Optionally, Global AvgPooling is used to compress the distance-speed feature map, the time-distance feature map, and the time-speed feature map.
[0114] Optionally, MLP is used for linear mapping to obtain attention weights, which are dot-producted with each feature map.
[0115] Optionally, a three-branch network feature fusion strategy based on the channel attention mechanism is adopted to adaptively learn the attention weights of each channel and different branches, so that the network can automatically and fully mine the useful information of the three feature maps across data domains, thereby improving the accuracy of human activity recognition.
[0116] It is understandable that by compressing the feature map corresponding to the three-dimensional data, the dimension of the data can be reduced, the complexity of the calculation can be reduced, and the most important information can be retained; using the attention mechanism for feature fusion, weights can be dynamically assigned to different feature maps, focusing on key information, reducing unnecessary calculations, and helping to adapt to different scenarios and changes, and improving the accuracy, robustness, and efficiency of action recognition and classification tasks.
[0117] Step 132: Encode the spatial angle feature corresponding to the three-dimensional data at each angle to obtain a spatial angle feature vector corresponding to the three-dimensional data at each angle.
[0118] Step 133: input the spatial angle feature vectors corresponding to the three-dimensional data at multiple angles into the Transformer layer to obtain the attention features corresponding to the three-dimensional data at multiple angles output by the Transformer layer.
[0119] Among them, the Transformer layer is an important component in the deep learning model. The Transformer layer consists of multiple sub-layers and can use the self-attention mechanism and feedforward neural network to effectively process sequence data and capture long-distance dependencies.
[0120] Step 134: input the attention features into the recognition layer to obtain the human action recognition results corresponding to the three-dimensional data of multiple angles output by the recognition layer.
[0121] In some embodiments, the recognition layer includes a multilayer perceptron (MLP) layer and a softmax classification layer.
[0122] Correspondingly, step 134 inputs the attention feature into the recognition layer to obtain the human motion recognition result corresponding to the three-dimensional data of multiple angles output by the recognition layer, including the following steps 1341, 1342 and 1343.
[0123] Step 1341: perform feature compression on the attention feature to obtain a one-dimensional attention feature.
[0124] Step 1342: Use the MLP layer to perform nonlinear mapping on the one-dimensional attention feature to obtain the processed one-dimensional attention feature.
[0125] Step 1343: Use the softmax classification layer to perform action recognition on the processed one-dimensional attention features to obtain action categories corresponding to the three-dimensional data at multiple angles.
[0126] It can be understood that by compressing high-dimensional attention features into one-dimensional representation, the complexity of calculation can be reduced while retaining the most critical information for action recognition; by using the MLP layer to perform nonlinear mapping on the one-dimensional attention features, it helps to capture more complex patterns and relationships and improve the accuracy of action recognition; by using the softmax classification layer to convert the processed one-dimensional attention features into probability distributions corresponding to the probabilities of different action categories, clear action classification results can be provided for the three-dimensional data at each angle.
[0127] Figure 7 A schematic diagram of a process for determining a deep learning model according to an embodiment of the present invention. Figure 7As shown, in some embodiments, the process of determining the deep learning model includes the following steps 710, step 720, step 730, step 740 and step 750.
[0128] Step 710: Obtain sample millimeter-wave radar echo data at multiple angles reflected by the human body.
[0129] Step 720: Process the sample millimeter-wave radar echo data at multiple angles to obtain sample three-dimensional data at multiple angles, where the sample three-dimensional data includes sample data in three dimensions: distance, time, and Doppler velocity.
[0130] Step 730: Determine human motion recognition result labels for sample three-dimensional data at multiple angles.
[0131] Step 740: Use the sample three-dimensional data at multiple angles as training samples and the human motion recognition result labels of the sample three-dimensional data at multiple angles as sample labels to train an initial deep learning model.
[0132] Step 750: Iteratively optimize the parameters of the initial deep learning model to obtain a deep learning model.
[0133] Optionally, training the initial deep learning model includes: inputting sample three-dimensional data at multiple angles into the initial deep learning model, and obtaining human motion prediction results corresponding to the sample three-dimensional data at multiple angles output by the initial deep learning model.
[0134] Optionally, the initial deep learning model includes an initial feature extraction layer, an initial Transformer layer, and an initial recognition layer.
[0135] Optionally, the sample three-dimensional data at each angle is input into the initial feature extraction layer to obtain the sample space angle features corresponding to the sample three-dimensional data at each angle output by the initial feature extraction layer; the sample space angle features corresponding to the sample three-dimensional data at each angle are encoded to obtain the sample space angle feature vector corresponding to the sample three-dimensional data at each angle; the sample space angle feature vectors corresponding to the sample three-dimensional data at multiple angles are input into the initial Transformer layer to obtain the sample attention features corresponding to the sample three-dimensional data at multiple angles output by the initial Transformer layer; the sample attention features are input into the initial recognition layer to obtain the human body motion prediction results corresponding to the sample three-dimensional data at multiple angles output by the initial recognition layer.
[0136] Optionally, the initial feature extraction layer includes an initial projection layer, an initial two-dimensional feature extraction layer and an initial feature fusion layer.
[0137] Optionally, the sample three-dimensional data of each angle is input into the initial projection layer to obtain a sample distance-speed two-dimensional graph, a sample time-distance two-dimensional graph and a sample time-speed two-dimensional graph corresponding to the sample three-dimensional data of each angle output by the initial projection layer.
[0138] Optionally, the sample distance-speed two-dimensional graph, the sample time-distance two-dimensional graph and the sample time-speed two-dimensional graph corresponding to the sample three-dimensional data at each angle are input into the initial two-dimensional feature extraction layer to obtain the sample distance-speed feature graph, the sample time-distance feature graph and the sample time-speed feature graph corresponding to the sample three-dimensional data at each angle output by the initial two-dimensional feature extraction layer.
[0139] Optionally, the sample distance-speed feature map, the sample time-distance feature map and the sample time-speed feature map corresponding to the sample three-dimensional data at each angle are input into the initial feature fusion layer to obtain the sample spatial angle features corresponding to the sample three-dimensional data at each angle output by the initial feature fusion layer.
[0140] It can be understood that by projecting the sample three-dimensional data containing the three dimensions of radar range-time-Doppler velocity onto the range-time plane, the range-Doppler velocity plane and the time-Doppler velocity plane respectively, and performing feature extraction, the sample distance-velocity feature map, the sample time-distance feature map and the sample time-velocity feature map are obtained, and the three feature maps are fused to realize the training of the deep learning model, which can not only fully extract key information and improve the performance of the deep learning model, but also reduce the complexity and cost of model training and improve the efficiency of model training.
[0141] Optionally, the initial recognition layer includes an initial MLP layer and an initial softmax classification layer.
[0142] Optionally, the sample attention features are feature compressed to obtain the sample one-dimensional attention features; the initial MLP layer is used to perform nonlinear mapping on the sample one-dimensional attention features to obtain the processed sample one-dimensional attention features; the initial softmax classification layer is used to perform action recognition on the processed sample one-dimensional attention features to obtain action category prediction results corresponding to the sample three-dimensional data at multiple angles.
[0143] It can be understood that by acquiring sample millimeter-wave radar echo data from multiple angles, a more comprehensive data view can be provided, enhancing the deep learning model's understanding of human movements; the sample three-dimensional data obtained through data processing can provide rich information, which helps the deep learning model capture complex motion patterns; and by iteratively optimizing the parameters of the initial deep learning model, the accuracy, robustness, and generalization ability of the deep learning model can be improved.
[0144] The radar human activity recognition device based on deep learning provided by the present invention is described below. The radar human activity recognition device based on deep learning described below and the radar human activity recognition method based on deep learning described above can be referenced to each other.
[0145] Figure 8 A schematic diagram of the structure of a radar human activity recognition device based on deep learning provided by an embodiment of the present invention, such as Figure 8 As shown, the radar human activity recognition device 800 based on deep learning includes:
[0146] The data acquisition unit 810 is used to collect millimeter wave radar echo data at multiple angles reflected by the human body;
[0147] A data processing unit 820 is used to process the millimeter wave radar echo data at multiple angles to obtain three-dimensional data at multiple angles, where the three-dimensional data includes data in three dimensions: distance, time, and Doppler velocity;
[0148] The action recognition unit 830 is used to input the three-dimensional data at multiple angles into a pre-built deep learning model to obtain the human action recognition results corresponding to the three-dimensional data at multiple angles output by the deep learning model;
[0149] Among them, the deep learning model is trained based on sample three-dimensional data from multiple angles and human motion recognition result labels of sample three-dimensional data from multiple angles.
[0150] Optionally, the deep learning model includes a feature extraction layer, a Transformer layer, and a recognition layer;
[0151] Correspondingly, the three-dimensional data at multiple angles are input into a pre-built deep learning model to obtain the human action recognition results corresponding to the three-dimensional data at multiple angles output by the deep learning model, including:
[0152] Input the three-dimensional data of each angle into the feature extraction layer, and obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature extraction layer;
[0153] Encode the spatial angle feature corresponding to the three-dimensional data of each angle to obtain the spatial angle feature vector corresponding to the three-dimensional data of each angle;
[0154] The spatial angle feature vectors corresponding to the three-dimensional data at multiple angles are input into the Transformer layer to obtain the attention features corresponding to the three-dimensional data at multiple angles output by the Transformer layer;
[0155] The attention features are input into the recognition layer to obtain the human action recognition results corresponding to the three-dimensional data of multiple angles output by the recognition layer.
[0156] Optionally, the feature extraction layer includes a projection layer, a two-dimensional feature extraction layer and a feature fusion layer;
[0157] Correspondingly, the three-dimensional data of each angle is input into the feature extraction layer, and the spatial angle features corresponding to the three-dimensional data of each angle output by the feature extraction layer are obtained, including:
[0158] Input the three-dimensional data of each angle into the projection layer, and obtain the distance-speed two-dimensional graph, time-distance two-dimensional graph and time-speed two-dimensional graph corresponding to the three-dimensional data of each angle output by the projection layer;
[0159] Inputting the distance-speed two-dimensional graph, time-distance two-dimensional graph and time-speed two-dimensional graph corresponding to the three-dimensional data of each angle into the two-dimensional feature extraction layer, obtaining the distance-speed feature graph, time-distance feature graph and time-speed feature graph corresponding to the three-dimensional data of each angle output by the two-dimensional feature extraction layer;
[0160] The distance-speed feature map, time-distance feature map and time-speed feature map corresponding to the three-dimensional data of each angle are input into the feature fusion layer to obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature fusion layer.
[0161] Optionally, the distance-speed feature map, the time-distance feature map, and the time-speed feature map corresponding to the three-dimensional data of each angle are input into the feature fusion layer to obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature fusion layer, including:
[0162] Compressing the distance-speed characteristic graph, the time-distance characteristic graph and the time-speed characteristic graph corresponding to the three-dimensional data of each angle to obtain the corresponding distance-speed characteristic compression graph, the time-distance characteristic compression graph and the time-speed characteristic compression graph;
[0163] The attention mechanism is used to obtain the attention weights of the distance-speed feature compression map, the time-distance feature compression map, and the time-speed feature compression map respectively;
[0164] Based on the distance-speed feature map, time-distance feature map and time-speed feature map corresponding to the three-dimensional data of each angle, as well as the attention weights of the distance-speed feature compression map, time-distance feature compression map and time-speed feature compression map, feature dot product is performed to obtain the spatial angle features corresponding to the three-dimensional data of each angle.
[0165] Optionally, the recognition layer includes a multi-layer perceptron MLP layer and a softmax classification layer;
[0166] Correspondingly, the attention features are input into the recognition layer to obtain the human action recognition results corresponding to the three-dimensional data of multiple angles output by the recognition layer, including:
[0167] Perform feature compression on the attention feature to obtain a one-dimensional attention feature;
[0168] The MLP layer is used to perform nonlinear mapping on the one-dimensional attention feature to obtain the processed one-dimensional attention feature;
[0169] The softmax classification layer is used to perform action recognition on the processed one-dimensional attention features to obtain the action categories corresponding to the three-dimensional data at multiple angles.
[0170] Optionally, the millimeter wave radar echo data at multiple angles reflected by the human body is collected, including:
[0171] An antenna array consisting of multiple receiving antennas is used to collect millimeter-wave radar echo data at multiple angles reflected by the human body;
[0172] Process the millimeter-wave radar echo data at multiple angles to obtain three-dimensional data at multiple angles, including:
[0173] Perform beamforming processing on millimeter-wave radar echo data at multiple angles to obtain three-dimensional radar data at multiple angles;
[0174] The three-dimensional radar data at multiple angles are processed by two-dimensional fast Fourier transform to obtain three-dimensional data at multiple angles.
[0175] Optionally, performing two-dimensional fast Fourier transform processing on the three-dimensional radar data at multiple angles to obtain three-dimensional data at multiple angles includes:
[0176] Perform one-dimensional fast Fourier transform on the three-dimensional radar data of each angle along the distance dimension to obtain distance information;
[0177] Perform one-dimensional fast Fourier transform on the three-dimensional radar data at each angle along the Doppler velocity dimension to obtain velocity information at different distances;
[0178] Based on the distance information and the speed information, transformed two-dimensional data is obtained, and based on the transformed two-dimensional data, three-dimensional data at each angle is obtained.
[0179] Optionally, the process of determining the deep learning model includes:
[0180] Obtain sample millimeter wave radar echo data at multiple angles reflected by the human body;
[0181] Processing the sample millimeter-wave radar echo data at multiple angles to obtain sample three-dimensional data at multiple angles, the sample three-dimensional data including sample data in three dimensions of distance, time and Doppler velocity;
[0182] Determine human action recognition result labels for sample three-dimensional data at multiple angles;
[0183] Using sample 3D data from multiple angles as training samples and human action recognition result labels of sample 3D data from multiple angles as sample labels, an initial deep learning model is trained;
[0184] The parameters of the initial deep learning model are iteratively optimized to obtain a deep learning model.
[0185] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the radar human activity recognition method based on deep learning, the method comprising: collecting millimeter wave radar echo data at multiple angles reflected by the human body; processing the millimeter wave radar echo data at multiple angles to obtain three-dimensional data at multiple angles, the three-dimensional data including data in three dimensions of distance, time and Doppler velocity; inputting the three-dimensional data at multiple angles into a pre-built deep learning model to obtain human action recognition results corresponding to the three-dimensional data at multiple angles output by the deep learning model; wherein the deep learning model is trained based on sample three-dimensional data at multiple angles and labels of human action recognition results of sample three-dimensional data at multiple angles.
[0186] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0187] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A radar human activity recognition method based on deep learning, characterized in that: include: Collect millimeter-wave radar echo data from multiple angles reflected by the human body; Processing the millimeter-wave radar echo data at the multiple angles to obtain three-dimensional data at the multiple angles, wherein the three-dimensional data includes data in three dimensions: distance, time, and Doppler velocity; Inputting the three-dimensional data at multiple angles into a pre-built deep learning model to obtain human motion recognition results corresponding to the three-dimensional data at multiple angles output by the deep learning model; The deep learning model is obtained by training based on sample three-dimensional data at multiple angles and labels of human action recognition results of the sample three-dimensional data at multiple angles; The deep learning model includes a feature extraction layer, a Transformer layer and a recognition layer; Correspondingly, the inputting the three-dimensional data of the multiple angles into a pre-built deep learning model to obtain the human action recognition results corresponding to the three-dimensional data of the multiple angles output by the deep learning model includes: Inputting the three-dimensional data of each angle into the feature extraction layer, and obtaining the spatial angle features corresponding to the three-dimensional data of each angle output by the feature extraction layer; Encoding the spatial angle feature corresponding to the three-dimensional data of each angle to obtain the spatial angle feature vector corresponding to the three-dimensional data of each angle; Inputting the spatial angle feature vectors corresponding to the three-dimensional data at multiple angles into the Transformer layer, and obtaining the attention features corresponding to the three-dimensional data at multiple angles output by the Transformer layer; Inputting the attention feature into the recognition layer to obtain the human action recognition results corresponding to the three-dimensional data of the multiple angles output by the recognition layer; The feature extraction layer includes a projection layer, a two-dimensional feature extraction layer and a feature fusion layer; Correspondingly, the inputting of the three-dimensional data of each angle into the feature extraction layer to obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature extraction layer includes: Input the three-dimensional data of each angle into the projection layer, and obtain a distance-speed two-dimensional graph, a time-distance two-dimensional graph, and a time-speed two-dimensional graph corresponding to the three-dimensional data of each angle output by the projection layer; Inputting the distance-speed two-dimensional graph, the time-distance two-dimensional graph and the time-speed two-dimensional graph corresponding to the three-dimensional data of each angle into the two-dimensional feature extraction layer, and obtaining the distance-speed feature graph, the time-distance feature graph and the time-speed feature graph corresponding to the three-dimensional data of each angle output by the two-dimensional feature extraction layer; The distance-speed feature graph, time-distance feature graph and time-speed feature graph corresponding to the three-dimensional data of each angle are input into the feature fusion layer to obtain the spatial angle features corresponding to the three-dimensional data of each angle output by the feature fusion layer.
2. The radar human activity recognition method based on deep learning according to claim 1 is characterized in that: The step of inputting the distance-speed feature graph, the time-distance feature graph, and the time-speed feature graph corresponding to the three-dimensional data of each angle into the feature fusion layer to obtain the spatial angle feature corresponding to the three-dimensional data of each angle output by the feature fusion layer includes: Compressing the distance-speed characteristic graph, the time-distance characteristic graph and the time-speed characteristic graph corresponding to the three-dimensional data of each angle to obtain corresponding distance-speed characteristic compression graph, time-distance characteristic compression graph and time-speed characteristic compression graph; Using an attention mechanism, the attention weights of the distance-speed feature compression map, the time-distance feature compression map, and the time-speed feature compression map are obtained respectively; Based on the distance-speed feature map, time-distance feature map and time-speed feature map corresponding to the three-dimensional data of each angle, and the attention weights of the distance-speed feature compression map, time-distance feature compression map and time-speed feature compression map, feature dot product is performed to obtain the spatial angle features corresponding to the three-dimensional data of each angle.
3. The radar human activity recognition method based on deep learning according to claim 1, characterized in that: The recognition layer includes a multi-layer perceptron MLP layer and a softmax classification layer; Correspondingly, the inputting the attention feature into the recognition layer to obtain the human action recognition results corresponding to the three-dimensional data of the multiple angles output by the recognition layer includes: Performing feature compression on the attention feature to obtain a one-dimensional attention feature; Using an MLP layer to perform nonlinear mapping on the one-dimensional attention feature to obtain a processed one-dimensional attention feature; A softmax classification layer is used to perform action recognition on the processed one-dimensional attention features to obtain action categories corresponding to the three-dimensional data at multiple angles.
4. The method for radar human activity recognition based on deep learning according to any one of claims 1 to 3, characterized in that: The collecting of millimeter wave radar echo data at multiple angles reflected by the human body includes: An antenna array consisting of multiple receiving antennas is used to collect millimeter-wave radar echo data at multiple angles reflected by the human body; The processing of the millimeter wave radar echo data at the multiple angles to obtain three-dimensional data at the multiple angles includes: Performing beamforming processing on the millimeter-wave radar echo data at the multiple angles to obtain three-dimensional radar data at the multiple angles; Perform two-dimensional fast Fourier transform processing on the three-dimensional radar data at the multiple angles to obtain the three-dimensional data at the multiple angles.
5. The method for radar human activity recognition based on deep learning according to claim 4, characterized in that: The performing two-dimensional fast Fourier transform processing on the three-dimensional radar data at the multiple angles to obtain the three-dimensional data at the multiple angles includes: Perform one-dimensional fast Fourier transform on the three-dimensional radar data of each angle along the distance dimension to obtain distance information; Perform one-dimensional fast Fourier transform on the three-dimensional radar data at each angle along the Doppler velocity dimension to obtain velocity information at different distances; Based on the distance information and the speed information, transformed two-dimensional data is obtained, and based on the transformed two-dimensional data, three-dimensional data at each angle is obtained.
6. The radar human activity recognition method based on deep learning according to claim 1, characterized in that: The process of determining the deep learning model includes: Obtain sample millimeter wave radar echo data at multiple angles reflected by the human body; Processing the sample millimeter-wave radar echo data at the multiple angles to obtain sample three-dimensional data at the multiple angles, wherein the sample three-dimensional data includes sample data in three dimensions: distance, time, and Doppler velocity; Determine human action recognition result labels of the sample three-dimensional data of the multiple angles; Using the sample three-dimensional data of the multiple angles as training samples and the human action recognition result labels of the sample three-dimensional data of the multiple angles as sample labels to train an initial deep learning model; Iteratively optimize the parameters of the initial deep learning model to obtain the deep learning model.
7. A radar human activity recognition device based on deep learning, characterized in that: The device is used to execute the radar human activity recognition method based on deep learning according to any one of claims 1 to 6, and the device includes: A data acquisition unit, used to collect millimeter-wave radar echo data at multiple angles reflected by the human body; A data processing unit, used to process the millimeter-wave radar echo data at multiple angles to obtain three-dimensional data at multiple angles, wherein the three-dimensional data includes data in three dimensions: distance, time, and Doppler velocity; An action recognition unit, configured to input the three-dimensional data at multiple angles into a pre-built deep learning model, and obtain a human action recognition result corresponding to the three-dimensional data at multiple angles output by the deep learning model; Among them, the deep learning model is trained based on sample three-dimensional data from multiple angles and human motion recognition result labels of the sample three-dimensional data from multiple angles.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the deep learning-based radar human activity recognition method as described in any one of claims 1 to 6 is implemented.
