A multi-channel radar lightweight human behavior recognition method based on hard classification decision

Abstract

The application discloses a multi-channel radar lightweight human behavior recognition method based on hard classification decision, applied to the technical field of behavior recognition, and aims at the problem of insufficient recognition performance of existing human behavior recognition methods; the application firstly collects multi-channel human behavior data collected based on a millimeter wave radar, processes relevant radar original echo signals through MTI technology, and then respectively performs STFT to obtain radar heat maps of human target distance dimensions, Doppler dimensions and the like; the obtained multi-channel radar heat maps are respectively input into multiple independent MobileNetV3 lightweight network models as data sets for classification recognition; subsequently, the multi-channel recognition results are determined through a set of algorithms based on hard classification decision, multi-channel human behavior information is fused, and finally, a human behavior discrimination label is obtained.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of behavior recognition, and particularly relates to a multi-channel radar lightweight human behavior recognition technology based on hard classification decision based on millimeter wave radar collection. BACKGROUND

[0002] Human behavior recognition technology aims to recognize and interpret human behavior through human behavior data obtained in various ways. With the continuous development of the field of artificial intelligence and the continuous maturity of the Internet of Things technology, human behavior recognition has become an increasingly important research topic in the fields of public safety, intelligent aging, body sensing game interaction, traffic regulation, etc. For human behavior recognition research, human behavior recognition technology using millimeter wave radar is increasingly favored in human behavior recognition technology because of its advantages such as all-weather monitoring, no need to wear, no privacy leakage, etc.

[0003] For human behavior recognition technology based on millimeter wave radar, there are currently several implementation methods: first, the method of applying logical judgment, threshold detection has a series of advantages such as simplicity and fast reaction processing speed, but the poor classification performance caused by false negatives and false positives is one of the biggest problems of this method, and the number and accuracy of thresholds also restrict the performance of the algorithm. Second, the method of applying traditional machine learning, which mainly relies on features extracted from sliding windows and classic machine learning algorithms (such as k-nearest neighbors, support vector machines, etc.), but traditional machine learning methods also have the disadvantages of tedious handcrafted feature extraction and performance constraints. Finally, the deep learning method that has gradually become popular in recent years, with more and more high-performance networks being applied to human behavior recognition, this classification method has gradually become the mainstream. Deep learning eliminates the tedious steps of feature extraction in traditional machine learning, and the original data set can be directly trained, but radar data cannot be trained with big data due to the small data set, so obtaining sufficient data set has become a problem to be solved, in addition, the data set is usually obtained from single-channel radar data set, which cannot well utilize the target feature information obtained from MIMO radar, finally, real-time recognition of HAR also puts higher requirements on lightweight hardware transplantation methods. SUMMARY

[0004] In view of the above technical problems, the present application provides a multi-channel radar lightweight human behavior recognition method based on hard classification decision. First, the multi-channel human behavior data collected based on the millimeter wave radar is collected, and the relevant radar original echo signal is processed by the MTI technology, and then STFT (Short-Time Fourier Transform) is performed to obtain the radar heat map of the human target distance dimension, Doppler dimension and other dimensions. The obtained multi-channel radar heat map is input into multiple independent MobileNetV3 lightweight network models as a data set for classification and recognition, and then the multi-channel recognition result is determined by a set of algorithms based on hard classification decision, so as to fuse the multi-channel human behavior information and obtain the final human behavior discrimination label.

[0005] The technical scheme adopted by the present application is as follows: a multi-channel radar lightweight human behavior recognition method based on hard classification decision, comprising:

[0006] S1, obtaining original radar echo data and pre-processing;

[0007] S2, performing fast Fourier transform on the pre-processed radar echo data in the distance dimension;

[0008] S3, drawing a heat map according to the distance dimension fast Fourier transform result of step S2 to obtain a human behavior information distance dimension heat map;

[0009] S4, finding the position of the target distance in the distance dimension, and performing Doppler dimension fast Fourier transform at the position;

[0010] S5, drawing a heat map according to the Doppler dimension fast Fourier transform result of step S4 to obtain a human behavior information Doppler dimension heat map;

[0011] S5, inputting the human behavior information distance dimension heat map and the human behavior information Doppler dimension heat map into each independent MobileNetV3 lightweight network model according to the channel to extract the features of each channel, and then performing deep convolution and point-by-point convolution on the features extracted from each channel to obtain the preliminary classification labels of each channel;

[0012] S6, processing the preliminary classification labels of each channel by using an algorithm based on hard classification decision to obtain the final classification label.

[0013] The beneficial effects of this invention are as follows: This invention proposes a lightweight human behavior recognition method based on hard classification decision using multi-channel radar. First, this method fully utilizes multi-channel radar data acquired by MIMO radar, fusing multi-channel data information through a hard classification decision-based algorithm. This avoids the stability fluctuations caused by accidental frame data deviations in traditional single-channel recognition algorithms, increasing the robustness and effectiveness of human behavior recognition. Simultaneously, the algorithm mechanism for main and secondary data label discrimination fully realizes the fusion of human behavior information from different dimensions, further improving the performance of human behavior recognition. Second, the multi-round discrimination mechanism ensures the performance of the human behavior recognition task while maintaining... The invention reduces computational resources for discrimination: human behaviors that are not easily confused can be identified in just one round of discrimination, while easily confused human behaviors are segmented and judged separately. This discrimination method increases the recognition performance of easily confused human behaviors while avoiding direct voting in each round, greatly reducing the overall computational resources for human behavior recognition. Finally, the discrimination architecture based on lightweight networks achieves excellent human behavior recognition and classification performance with low computational cost. Its fast processing speed proves that it can be applied to real-time detection of related devices, proving to be a feasible lightweight hardware migration recognition method. Attached Figure Description

[0014] Figure 1 This is a flowchart of a lightweight human behavior recognition method based on hard classification decision using multi-channel radar.

[0015] Figure 2 This is a schematic diagram of the experimental scenario model.

[0016] Figure 3 This is a diagram of the actual experimental scenario.

[0017] Figure 4 This is a schematic diagram of the original three-dimensional data matrix obtained after preprocessing.

[0018] Figure 5 A distance-dimensional radar heatmap for fall behavior.

[0019] Figure 6 Doppler radar heatmap of fall behavior.

[0020] Figure 7 A schematic diagram of the overall process for obtaining behavior recognition classification labels.

[0021] Figure 8 This is a schematic diagram illustrating the process of obtaining the final classification label using an algorithm based on hard classification decision.

[0022] Figure 9 The final human behavior recognition performance confusion matrix. Detailed Implementation

[0023] To facilitate the technical content of the present application for those skilled in the art to understand, the content of the present application is further explained below in conjunction with the drawings.

[0024] The simulation examples mainly perform multi-classification on 6 common human behaviors, including waving, standing up, sitting down, falling down, getting up and lying down. In order to increase the robustness of the recognition result, the present application collects the target behaviors at different angles and distances for all behavior data collection. The specific experimental scene model is shown in Figure 2 The experimental site is relatively open, and various human behaviors are collected in a 5*5m square, and the radar distance from the ground is 2.8m. The actual experimental scene diagram is shown in Figure 3 .

[0025] As shown in Figure 1 , the processing process of the present application includes the following steps:

[0026] Step 1: Obtain and preprocess the original radar echo data

[0027] The corresponding beat signal is obtained using MIMO (Multiple-Input Multiple-Output) millimeter wave radar, and after being digitized by orthogonal ADC (Analog-to-Digital Converter) and discrete sampling, it is stored as a 256*12800*12 size original three-dimensional data matrix in a specific order, where 256 is the radar frame sampling point number N S , 12800 is the total chirp number N c in the sampling data frame, and 12 is the sampling data channel number M. The original three-dimensional data matrix obtained after preprocessing is shown in Figure 4 .

[0028] Then, for the original three-dimensional data obtained for each channel, MTI (Moving Target Indicator, moving target indicator) intra-frame cancellation is performed:

[0029] MTI (m,k) =TR (m,k) -TR (m+j,k)

[0030] In the formula, MTI (m,k) is the radar data after MTI intra-frame cancellation, j represents the continuous frequency modulation span of each cancellation, and after MTI intra-frame cancellation processing, the human behavior information that basically suppresses the scene static clutter can be obtained.

[0031] Step 2: Obtain the distance dimension and Doppler dimension heat map

[0032] After obtaining the radar 3D data matrix for basic suppression of static clutter in the scene, a fast Fourier transform is first performed in the range dimension:

[0033]

[0034] In the formula, ||·|| 2 TR represents the square of the modulus value. (m,k) This represents the amplitude of the m-th chirp at the digital frequency point 2πk / N after discrete Fourier transform, where N is the value of the chirp. S N represents the number of sampling points in the radar frame, and W represents the number of discrete Fourier transform points. u S is a pre-defined Hamming window function. (n-u,m) This represents the data of the null-th sampling point of the m-th chirp. n is the number of sampling points, and u is the integration variable. After processing, each channel yields a 256*12800 Range-FFT matrix, where 256 represents the number of Range-FFT sampling points, and 12800 represents the total number of chirps in the radar sampling data frame. Plotting the Range-FFT matrix as a heatmap yields the distance-dimensional heatmap of human behavior information. The distance-dimensional radar heatmap of fall behavior is shown below. Figure 5 As shown.

[0035] Then, the target's distance location is found in the distance dimension, and a Doppler-dimensional FFT (fast Fourier transform) is performed at that location. For the Range-FFT matrix after cancellation and suppression of static clutter, the data for each frame is summed row by row, and the maximum value of each column is searched. The index of the maximum value is regarded as the target location within that frame. Then, a Doppler-dimensional FFT is performed at the target location in each frame.

[0036]

[0037] VR in the formula (d) This represents the amplitude at the digital frequency 2πd / N point after the discrete Fourier transform on the k-th row of the time-distance image, where N represents the number of discrete Fourier transform points, M is the window length of the Fourier transform, and W... u For a predefined Hamming window function, TR (n-u,m) This represents the time-distance data in the k-th row of the frame. After processing, a 256*100 velocity data matrix is ​​obtained, where 256 represents the number of Doppler-FFT sampling points and 100 represents the total number of radar frames in one detection. Plotting the Doppler-FFT matrix as a heatmap yields a Doppler heatmap of human behavior information, thus providing preliminary information on the distance and velocity of human behavior. An example of a Doppler radar heatmap of fall behavior is shown below. Figure 6 As shown.

[0038] Step 3: independent lightweight network model classification identification obtains preliminary classification label

[0039] The total flowchart for obtaining the behavior recognition classification label is shown in Figure 7

[0040] First, for the obtained distance dimension heat map and Doppler heat map, we respectively put them into independent MobileNetV3 lightweight network models according to their channel sources for feature extraction: the distance item and the time-frequency spectrum of each channel are respectively subjected to scaling and normalization, and then subjected to deep convolution and point-by-point convolution, the process is as follows:

[0041] Y dw =∑ k ∑ i,j w k,i,j ·X i,j ;Y pw =∑ k ∑ c w k,c ·Y dw,c

[0042] In the formula, Y dw represents the deep convolution result, w k,i,j is the deep convolution corresponding weight, X i,j is the input distance item and time-frequency spectrum image pixel; Y pw represents the point-by-point convolution result, w k,c is the point-by-point convolution corresponding weight, Y dw,c is the deep convolution result obtained before, and the subscript c represents the channel dimension. The obtained feature vector is then subjected to a lightweight bottleneck layer, and after fusing the feature maps of different scales, it is subjected to an FC connection layer (Fully Connected Layer) and an activation function, finally obtaining the discrimination probability of each classification, selecting the class with the highest appearance probability as the preliminary classification label, and taking the label of different channels obtained based on the distance item as the main label Q1, and taking the label of different channels obtained based on the time-frequency spectrum as the auxiliary label Q2.

[0043] Step 4: obtaining the final classification label based on the hard classification decision algorithm

[0044] After obtaining the preliminary classification label based on the independent lightweight network model, the application obtains the final classification label based on a hard classification decision algorithm, and the flowchart for obtaining the final classification label based on the hard classification decision algorithm is shown in Figure 8 , and the specific algorithm is as follows:

[0045] ​In the first round of main decision, the obtained discriminant main label Q1 of each channel is subjected to non-probability model decision, and the preliminary decision vector with the highest occurrence frequency among the decision vectors representing different target behaviors is taken as the final decision vector. In this embodiment, a decision vector decision threshold w1 is set. If the difference between the two decision vectors corresponding to the two target behaviors with the highest occurrence frequency is less than the threshold w1, it is determined that the human behaviors represented by the several labels are easily confused behaviors, and the decision enters the first round of secondary decision; otherwise, the final decision vector is output, and the decision is ended.

[0046] The value of the threshold is explained as follows. For example, the decision threshold w1 is set to 1. In the classification behavior, the two behaviors with the highest decision frequency are sitting with 6 channels and falling with 3 channels, and the difference is greater than the decision threshold w1. It can be determined that the final behavior is sitting, and the decision is ended. Conversely, if the two behaviors with the highest decision frequency are sitting with 5 channels and falling with 4 channels, and the difference between the two behaviors is equal to the decision threshold w1, it is proved that the two behaviors are easily confused behaviors, and the next round of decision is needed. In this embodiment, the values of w1 and w2 are 1, and in actual application, the values can be set according to the needs.

[0047] In the first round of secondary decision, only the easily confused behaviors recorded in the first round of main decision are subjected to non-probability model decision on the obtained discriminant auxiliary label Q2 of each channel. Similarly, the preliminary decision vector with the highest occurrence frequency among the decision vectors representing different target behaviors is taken as the final decision vector. In this embodiment, a decision vector decision threshold w2 is set. If the number of hard classification decision labels of the final decision vector obtained and other preliminary decision vectors of different behaviors is less than the threshold w2, it is determined that the auxiliary label Q2 cannot completely discriminate the above-mentioned easily confused human behaviors, and the decision enters the final decision.

[0048] In the final decision, the recorded easily confused behaviors are subjected to independent lightweight network model classification recognition based on different channels again. However, unlike the first round of discrimination, the test set discrimination category of this embodiment is only marked as the recorded easily confused behaviors, and in the final decision, the obtained discriminant main label Q1 and discriminant auxiliary label Q2 are directly subjected to voting decision, and the behavior type with the most votes is taken as the final classification label of the final discrimination.

[0049] The final human behavior recognition performance confusion matrix is shown in Table 1. Figure 9 As can be seen from Table 1, the method has achieved very good recognition effect for the simulation example of 6 human behaviors.

[0050] In order to clearly show the superiority of the algorithm of the present application, the present embodiment carries out experiments on the classification results based on the MobileNetV3 network architecture for single channel. In order to avoid the influence of insufficient data quantity on the results of single channel ablation experiment, we integrate three channel data as a training set and a test set for single channel experiment, so as to ensure that the training data quantity of multi-channel experiment and single channel experiment is the same. The average accuracy comparison of multi-channel classification and single channel classification based on hard classification algorithm decision is shown in Table 1. We can see that, under the condition of the same training data quantity, the multi-channel classification based on hard classification algorithm decision not only fully integrates the target behavior information, but also has obvious improvement in behavior recognition accuracy, and effectively enhances the recognition robustness.

[0051] Table 1 Average accuracy comparison of multi-channel classification and single channel classification based on hard classification algorithm decision

[0052] Channel information obtained by the MobineNetV3 model Average accuracy 3-channel mixing 97.05％ Channel 1 94.04％ Channel 8 94.52％ Channel 11 93.52％

[0053] At the same time, we selected the hot GhostNet and EfficientNetv2-s network structure models in lightweight network, the representative network ResNet18 in traditional CNN (Convolutional Neural Network, Convolutional Neural Network), and the ViT (Vision Transformer) network combined with transformer structure for comparison test. Table 2 shows the comparison of average training accuracy, parameter quantity and FLOPs (Floating Point Operations Per Second, Floating Point Operations Per Second) of each network model. It can be seen that MobileNetV3 realizes higher classification accuracy with very small operation cost, and has unique advantages in the scene of radar equipment embedding and small data processing in the experiment.

[0054] Table 2 Comparison of average training accuracy, parameter quantity and FLOPs of each network model

[0055] Network model Average accuracy Parameter amount FLOPs MobileNetV3-s 97.05％ 1.62M 0.061G GhostNet 96.75％ 4.40M 0.22G EfficientNetv2-s 97.42％ 20.19M 2.89G ResNet18 97.13％ 11.2M 1.82G ViT 95.23％ 86.5M 16.86G

[0056] Those skilled in the art will appreciate that the embodiments described herein are presented for the purpose of helping the reader to understand the principles of the present application, and should be understood as not limiting the scope of protection of the present application to such specific statements and embodiments. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the scope of claims of the present application.

Claims

1. A method for multi-channel radar lightweight human behavior recognition based on hard classification decision, characterized in that, The method comprises the following steps: S1, obtaining original radar echo data and preprocessing; S2, performing fast Fourier transform on the radar echo data in the distance dimension after preprocessing; S3, drawing a heat map according to the result of the fast Fourier transform in the distance dimension in step S2 to obtain a human behavior information distance dimension heat map; S4, finding the position of the target distance in the distance dimension, and performing fast Fourier transform in the Doppler dimension at the position; S5, drawing a heat map according to the result of the fast Fourier transform in the Doppler dimension in step S4 to obtain a human behavior information Doppler dimension heat map; S6, inputting the human behavior information distance dimension heat map and the human behavior information Doppler dimension heat map into each independent MobileNetV3 light weight network model according to the channel to extract features in each channel, and then performing deep convolution and point-by-point convolution on the features extracted in each channel to obtain preliminary classification labels of each channel; The preliminary classification labels in step S6 include main labels and secondary labels, the main labels are labels obtained according to the human behavior information distance dimension heat map, and the secondary labels are labels obtained according to the human behavior information Doppler dimension heat map; S7, processing the preliminary classification labels of each channel by using an algorithm based on hard classification decision to obtain final classification labels; step S7 specifically comprises: S71, according to the main labels corresponding to each channel, counting the number of channels in which each type of human behavior appears; if the difference between the number of channels of the two types of human behaviors with the largest number of channels is greater than a set threshold, the type of human behavior with the largest number of channels is output as the final classification label; Otherwise, step S72 is performed; S72, according to the secondary labels corresponding to each channel, counting the number of channels in which each type of human behavior appears; if the difference between the number of channels of the two types of human behaviors with the largest number of channels is greater than a set threshold, the type of human behavior with the largest number of channels is output as the final classification label; Otherwise, step S73 is performed; S73, for the two types of human behaviors with the largest number of channels in step S72, the processing process of step S6 is performed again; and the discriminant main labels and discriminant secondary labels of each channel are directly used for voting decision, and the type of behavior with more votes is taken as the final classification label of the final discrimination.

2. The method according to claim 1, wherein, The preprocessing in step S1 specifically comprises: performing moving target display intra-frame cancellation processing on the original radar echo data.

3. The method according to claim 2, wherein, The process of finding the position of the target distance in the distance dimension in step S4 is: the result obtained after step S2 is denoted as a Range-FFT matrix, the maximum value of each column is searched after summing each frame of data of the Range-FFT matrix by row, and the index of the maximum value is regarded as the position of the target in the frame.

Images