Multi-channel radar lightweight human body behavior recognition method based on hard classification judgment

Abstract

The invention discloses a multi-channel radar lightweight human body behavior recognition method based on hard classification judgment, which is applied to the technical field of behavior recognition and aims at the problem of insufficient recognition performance of the existing human body behavior recognition method. The method comprises the following steps: firstly, collecting multi-channel human body behavior data acquired based on a millimeter wave radar, processing related radar original echo signals through an MTI technology, and then respectively carrying out STFT to obtain radar heat maps of dimensions such as a human body target distance dimension and a Doppler dimension; the obtained multi-channel radar heat map serves as a data set to be input into a plurality of independent MobileNetV3 lightweight network models for classification and recognition, then a multi-channel recognition result is judged through an algorithm based on hard classification judgment, multi-channel human body behavior information is fused, and a final human body behavior judgment label is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of behavior recognition, and in particular relates to a multi-channel radar lightweight human behavior recognition technology based on hard classification judgment under millimeter wave radar acquisition. Background Art

[0002] Human behavior recognition technology aims to identify and interpret human behavior through human behavior data obtained in various ways. With the continuous development of artificial intelligence and the continuous maturity of the Internet of Things technology, human behavior recognition has become a topic worthy of further research in the fields of public safety, smart elderly care, somatosensory game interaction, traffic supervision, etc. As for the research on human behavior recognition, human behavior recognition technology using millimeter wave radar is becoming more and more popular in human behavior recognition technology because of its advantages such as all-weather monitoring, no need to wear, and no privacy leakage.

[0003] As for the human behavior recognition technology based on millimeter wave radar, there are mainly the following implementation methods: First, the method of applying logical judgment. Threshold detection certainly has a series of advantages such as simplicity and fast response processing speed, but the poor classification performance caused by false negatives and false positives is one of the biggest problems restricting the method. The number and accuracy of thresholds also restrict the performance of the algorithm. Secondly, the method of applying traditional machine learning mainly relies on features extracted from sliding windows and classic machine learning algorithms (such as k-nearest neighbors, support vector machines, etc.), but traditional machine learning methods also have the shortcomings of cumbersome manual feature extraction and performance constraints. Finally, the deep learning method that has become increasingly popular recently. With more and more high-performance networks being applied to human behavior recognition, this classification method has gradually become mainstream. Deep learning eliminates the tedious steps of extracting features in traditional machine learning, and the original data set can be directly trained. However, due to the small number of radar data sets, training cannot be performed on big data, so obtaining sufficient data sets has also become a problem to be solved. In addition, data sets are usually obtained from single-channel radar data sets, and MIMO radar cannot be used to obtain target feature information. Finally, real-time recognition of HAR also puts higher requirements on lightweight hardware transplantation methods. Summary of the invention

[0004] In view of the above technical problems, the present invention proposes a multi-channel radar lightweight human behavior recognition method based on hard classification judgment. First, the multi-channel human behavior data collected by the millimeter wave radar is collected, and the relevant radar original echo signal is processed by MTI technology, and then STFT (Short-Time Fourier Transform) is performed to obtain radar heat maps of human target distance dimension, Doppler dimension and other dimensions. The obtained multi-channel radar heat map is used as a data set and input into multiple independent MobileNetV3 lightweight network models for classification and recognition. Then, the multi-channel recognition results are judged by a set of algorithms based on hard classification judgment, so that the multi-channel human behavior information is fused to obtain the final human behavior discrimination label.

[0005] The technical solution adopted by the present invention is: a multi-channel radar lightweight human behavior recognition method based on hard classification judgment, comprising:

[0006] S1, obtaining the original radar echo data and preprocessing it;

[0007] S2, performing fast Fourier transform on the preprocessed radar echo data in the distance dimension;

[0008] S3, drawing a heat map according to the distance dimension fast Fourier transform result of step S2, and obtaining a distance dimension heat map of human behavior information;

[0009] S4, finding the position of the target distance in the distance dimension, and performing a Doppler dimension fast Fourier transform at the position;

[0010] S5, drawing a heat map according to the Doppler fast Fourier transform result of step S4 to obtain a Doppler heat map of human behavior information;

[0011] S5, inputting the distance dimension heat map of human behavior information and the Doppler dimension heat map of human behavior information into each independent MobileNetV3 lightweight network model according to the channel to extract the features of each channel, and then performing deep convolution and point-by-point convolution on the features extracted from each channel to obtain the preliminary classification labels of each channel;

[0012] S6. Use an algorithm based on hard classification decision to process the preliminary classification labels of each channel to obtain the final classification labels.

[0013] Beneficial effects of the invention: The invention proposes a multi-channel radar lightweight human behavior recognition method based on hard classification judgment. First, the method makes full use of the multi-channel radar data obtained by the MIMO radar, and fuses the multi-channel data information through an algorithm based on hard classification judgment, thereby avoiding the judgment stability fluctuation caused by the accidental frame data deviation of the traditional single-channel recognition algorithm, and increasing the robustness and robustness of human recognition; at the same time, the algorithm mechanism for distinguishing the primary and secondary data labels fully realizes the fusion of human behavior information of different dimensions, and further improves the performance of human behavior recognition; secondly, the multi-round judgment mechanism ensures the performance of human behavior recognition tasks while ensuring the performance of human behavior recognition tasks. The discrimination computing resources are reduced: human behaviors that are not easily confused only need one round of discrimination to be identified, while human behaviors that are easily confused are segmented and then subjected to final discrimination separately. This discrimination method increases the recognition performance of human behaviors that are easily confused while avoiding direct voting judgment in each round, greatly reducing the discrimination computing resources of the overall human behavior recognition task; finally, the discrimination architecture based on the lightweight network of the present invention achieves relatively good human behavior recognition and classification performance under the condition of using a relatively small computational cost, and its faster processing rate proves that it can be applied to real-time detection of related equipment, and is proved to be a feasible lightweight hardware migration recognition method. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is the overall flow chart of the multi-channel radar lightweight human behavior recognition method based on hard classification judgment.

[0015] Figure 2 Schematic diagram of the experimental scene model.

[0016] Figure 3 This is the actual scene diagram of the experiment.

[0017] Figure 4 Schematic diagram of the original three-dimensional data matrix obtained after preprocessing.

[0018] Figure 5 Distance-dimensional radar heat map for falling behavior.

[0019] Figure 6 Doppler radar heat map of falling behavior.

[0020] Figure 7 Schematic diagram of the overall process of obtaining behavior recognition classification label discrimination.

[0021] Figure 8 Schematic diagram of the process of obtaining the final classification label for the algorithm based on hard classification decision.

[0022] Fig. 9 Confusion matrix for the final human action recognition performance. DETAILED DESCRIPTION

[0023] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0024] The simulation example mainly performs multi-classification on six common human behaviors, including waving, standing up, sitting down, falling, getting up and lying down. In order to increase the robustness of the recognition results, the present invention collects target behaviors at different angles and distances for all behavior data collection. The specific experimental scene model is as follows: Figure 2 As shown in the figure, the experimental site is relatively open, and various human behaviors are collected in a 5*5m square. The radar is 2.8m above the ground. Figure 3 shown.

[0025] like Figure 1 As shown, the processing process of the present invention includes the following steps:

[0026] Step 1: Acquisition and preprocessing of raw radar echo data

[0027] The corresponding beat signal is obtained using MIMO (Multiple-Input Multiple-Output) millimeter-wave radar, which is digitized and discretely sampled by an orthogonal ADC (Analog-to-Digital Converter) and stored in a 256*12800*12 original three-dimensional data matrix in a specific order, where 256 is the number of radar frame sampling points N. S , 12800 is the total number of chirps in the sampled data frame N c , 12 is the number of sampling data channels M. The original three-dimensional data matrix obtained after preprocessing is as follows Figure 4 shown.

[0028] Then, for the original 3D data obtained from each channel, MTI (Moving Target Indicator) intra-frame cancellation is performed:

[0029] MTI (m,k) =TR (m,k) -TR (m+j,k)

[0030] Where, MTI (m,k) is the radar data after MTI intra-frame cancellation, j represents the continuous frequency modulation span of each cancellation. After MTI intra-frame cancellation processing, the human behavior information that basically suppresses the static clutter of the scene can be obtained.

[0031] Step 2: Obtain heatmaps in distance and Doppler dimensions

[0032] After obtaining the radar 3D data matrix that basically suppresses the static clutter of the scene, first perform a fast Fourier transform in the distance dimension:

[0033]

[0034] In the formula, ||·|| 2 Indicates the square of the modulus value, TR (m,k) represents the amplitude of the mth chirp after discrete Fourier transform at the digital frequency 2πk / N, N S is the number of radar frame sampling points, N is the number of discrete Fourier transform points, W u is a preset Hamming window function, S (n-u,m) Represents the data of the nuth sampling point of the mth chirp. n is the sampling point, and u is the integral variable. After processing, each channel obtains a 256*12800 Range-FFT matrix, where 256 is the number of sampling points of the Range-FFT and 12800 is the total number of chirps in the radar sampling data frame. The Range-FFT matrix is ​​plotted in the form of a heat map, and the distance dimension heat map of human behavior information is obtained. The distance dimension radar heat map of falling behavior is as follows Figure 5 shown.

[0035] Then find the position of the target distance in the distance dimension, and then perform Doppler dimension FFT (fast Fourier transform) at this position. For the Range-FFT matrix that suppresses static clutter after cancellation, sum each frame of data by row, search for the maximum value of each column by column, and the maximum value index is regarded as the target position in the frame. Then perform Doppler dimension FFT at the target position in each frame:

[0036]

[0037] VR (d) It means that after the k-th row of the time-distance image is subjected to discrete Fourier transform, the amplitude at the digital frequency 2πd / N point is obtained, where N is the number of discrete Fourier transform points, M is the window length of the Fourier transform, and W is the value of the digital frequency 2πd / N. u is a preset Hamming window function, TR (n-u,m) Represents the time distance data of the kth row on the Frame. After processing, a 256*100 speed data matrix is ​​obtained, where 256 is the number of Doppler-FFT sampling points and 100 is the total number of radar detection frames. The Doppler-FFT matrix is ​​plotted in the form of a heat map, and the Doppler dimension heat map of human behavior information is obtained, and then the distance and speed information of human behavior are preliminarily obtained. The Doppler dimension radar heat map of falling behavior is shown in the figure. Figure 6 shown.

[0038] Step 3: Independent lightweight network model classification and recognition to obtain preliminary classification labels

[0039] The overall process diagram for obtaining behavior recognition classification label discrimination is as follows Figure 7 shown.

[0040] First, we put the obtained distance dimension heat map and Doppler dimension heat map into independent MobileNetV3 lightweight network models according to their channel sources for feature extraction: the distance terms and time-frequency spectrum of each channel are scaled and normalized, and then subjected to depth convolution and point-by-point convolution respectively. The process is as follows:

[0041] Y dw =∑ k ∑ i,j w k,i,j ·X i,j ; Y pw =∑ k ∑ c w k,c ·Y dw,c

[0042] Where Y dw represents the result of deep convolution, w k,i,j is the weight corresponding to the depth convolution, X i,j is the input distance term and the pixel of the time-frequency spectrum; Y pw Represents the point-by-point convolution result, w k,c is the weight corresponding to the point-by-point convolution, Y dw,c is the result of the deep convolution obtained above, and the subscript c represents the channel dimension. The feature vector obtained later passes through a lightweight bottleneck layer, and after the feature maps of different scales are fused, it passes through the FC connection layer (Fully Connected Layer) and the activation function, and finally obtains the discrimination probability of each classification. The category with the highest probability of occurrence is selected as the preliminary classification label, and the labels of different channels obtained based on the distance term are used as the main label Q 1 , the labels of different channels obtained based on the time-spectrum graph are used as auxiliary labels Q 2 .

[0043] Step 4: Obtain the final classification label based on the algorithm decision of hard classification

[0044] After obtaining the preliminary classification labels based on the independent lightweight network model, the present invention applies an algorithm based on hard classification judgment to obtain the final classification labels. The flowchart of the algorithm based on hard classification judgment to obtain the final classification labels is as follows: Figure 8 As shown, the specific algorithm is as follows:

[0045] In the first round of main judgment, the main labels Q of each channel are obtained. 1Implement non-probabilistic model judgment, and use the preliminary judgment vector with the largest number of occurrences in the judgment vectors representing different target behaviors as the final judgment vector, wherein the present invention sets a judgment threshold w for the judgment vector 1 , if the difference between the decision vectors of the two target behaviors with the largest number of occurrences is less than the threshold w 1 , the human behaviors represented by these labels are judged to be easily confused behaviors, and the judgment enters the first round of sub-judgment; otherwise, the final judgment vector is output and the judgment ends.

[0046] Explanation of the threshold value: For example, the decision threshold w is set 1 is 1. The two behaviors with the most judgments in the algorithm in the classification behavior are sitting down with 6 channels and falling down with 3 channels. The difference between them is greater than the judgment threshold w 1 , it can be determined that the final behavior is sitting down, and the judgment ends; on the contrary, if the two most common behaviors are sitting down, 5 channels, and falling down, 4 channels, the difference between the two is equal to the judgment threshold w 1 , it proves that the two are easily confusing behaviors and the next round of judgment is needed. 1 、w 2 The value is 1. In actual applications, the value can be set as needed.

[0047] In the first round of secondary judgment, only the easily confusing behaviors recorded in the first round of primary judgment are used to obtain the auxiliary labels Q of each channel. 2 Implementing non-probabilistic model judgment, the initial judgment vector with the largest number of occurrences in the judgment vectors representing different target behaviors is also used as the final judgment vector, where a judgment threshold w of the discriminant vector is set. 2 , if the final decision vector obtained is different from the initial decision vector of other decisions, the number of hard classification decision labels is less than the threshold w 2 , then determine the auxiliary label Q 2 It is also impossible to fully distinguish the above-mentioned easily confused human behaviors, and the judgment becomes final.

[0048] In the final judgment, for the recorded easily confusing behaviors, a new classification and identification based on independent lightweight network models of different channels is performed. However, unlike the first round of discrimination, the discrimination category of the test set of the present invention is only marked as the recorded easily confusing behaviors, and the discrimination main label Q of each channel is obtained in the final judgment. 1 and discriminative auxiliary labels Q 2 Directly use voting judgment, and the behavior type with the most votes will be used as the final classification label for the final judgment.

[0049] The final human behavior recognition performance confusion matrix is ​​as follows Fig. 9As shown in Figure 2, it can be seen that this method has achieved very good recognition results for the six human behaviors in the simulation example.

[0050] In order to clearly demonstrate the superiority of the algorithm of the present invention, this embodiment experiments on the classification results based on the MobileNetV3 network architecture in a single channel. In order to avoid the influence of insufficient data on the results of the single-channel ablation experiment, we use the three-channel data as a training set and a test set for a single-channel experiment, thereby ensuring that the amount of training data for the multi-channel experiment is the same as that for the single-channel experiment. The comparison of the average accuracy of multi-channel classification and single-channel classification based on the hard classification algorithm is shown in Table 1. We can see that when the amount of training data is the same, the multi-channel classification based on the hard classification algorithm not only fully integrates the target behavior information, but also has a significant improvement in the accuracy of behavior recognition, and effectively enhances the recognition robustness.

[0051] Table 1 Comparison of average accuracy of multi-channel classification and single-channel classification based on hard classification algorithm judgment

[0052] Channel information obtained by the MobineNetV3 model Average accuracy 3-channel mixing 97.05％ Channel 1 94.04％ Channel 8 94.52％ Channel 11 93.52％

[0053] At the same time, we selected the popular GhostNet and EfficientNetv2-s network structure models in lightweight networks, the representative network ResNet18 in traditional CNN (Convolutional Neural Network), and the ViT (Vision Transformer) network combined with the transformer structure for comparative experiments. Table 2 shows the average training accuracy, parameter quantity, and FLOPs (Floating Point Operations Per Second) comparison of each network model. It can be seen that MobileNetV3 achieves a high classification accuracy with extremely low computing cost, which has unique advantages in the scenarios of radar device embedding and small data processing in this experiment.

[0054] Table 2 Comparison of average training accuracy, parameter count and FLOPs of each network model

[0055] Network Model Average accuracy Parameter quantity FLOPs MobileNetV3-s 97.05％ 1.62M 0.061G GhostNet 96.75％ 4.40M 0.22G EfficientNetv2-s 97.42％ 20.19M 2.89G ResNet18 97.13％ 11.2M 1.82G ViT 95.23％ 86.5M 16.86G

[0056] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A multi-channel radar lightweight human behavior recognition method based on hard classification judgment, characterized in that: include: S1, obtaining the original radar echo data and preprocessing it; S2, performing fast Fourier transform on the preprocessed radar echo data in the distance dimension; S3, drawing a heat map according to the distance dimension fast Fourier transform result of step S2, and obtaining a distance dimension heat map of human behavior information; S4, finding the position of the target distance in the distance dimension, and performing a Doppler dimension fast Fourier transform at the position; S5, drawing a heat map according to the Doppler fast Fourier transform result of step S4 to obtain a Doppler heat map of human behavior information; S6, inputting the distance dimension heat map of human behavior information and the Doppler dimension heat map of human behavior information into each independent MobileNetV3 lightweight network model according to the channel to extract the features of each channel, and then performing deep convolution and point-by-point convolution on the features extracted from each channel to obtain the preliminary classification labels of each channel; S7. Use an algorithm based on hard classification decision to process the preliminary classification labels of each channel to obtain the final classification labels.

2. According to the multi-channel radar lightweight human behavior recognition method based on hard classification judgment according to claim 1, it is characterized in that: The preprocessing described in step S1 specifically includes: performing moving target display frame cancellation processing on the original radar echo data.

3. The multi-channel radar lightweight human behavior recognition method based on hard classification judgment according to claim 2 is characterized in that: The process of finding the target distance position in the distance dimension in step S4 is as follows: the result obtained after processing in step S2 is recorded as the Range-FFT matrix, and the data of each frame of the Range-FFT matrix is ​​summed up row by row, and then the maximum value of each column is searched column by column. The index of the maximum value is regarded as the position of the target in the frame.

4. The multi-channel radar lightweight human behavior recognition method based on hard classification judgment according to claim 3 is characterized in that: The preliminary classification labels in step S6 include a main label and a sub-label, the main label is a label obtained according to the distance dimension heat map of human behavior information, and the sub-label is a label obtained according to the Doppler dimension heat map of human behavior information.

5. The multi-channel radar lightweight human behavior recognition method based on hard classification judgment according to claim 4 is characterized in that: Step S7 is specifically as follows: S71, according to the main label corresponding to each channel, counting the number of channels in which each type of human behavior appears; if the difference between the number of channels of the two types of human behaviors with the largest number of channels is greater than a set threshold, outputting the type of human behavior with the largest number of channels as the final classification label; Otherwise, execute step S72; S72, according to the secondary labels corresponding to each channel, counting the number of channels in which each type of human behavior appears; if the difference between the number of channels of the two types of human behaviors with the largest number of channels is greater than a set threshold, outputting the type of human behavior with the largest number of channels as the final classification label; Otherwise, execute step S73; S73. For the two human behaviors with the largest number of channels in step S72, the processing of step S6 is repeated; and the main discrimination labels and auxiliary discrimination labels of each channel are directly judged by voting, and the behavior type with the largest number of votes will be used as the final classification label for the final judgment.

Images