Human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network

Abstract

The present invention relates to the technical field of human behavior recognition, and in particular discloses a human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-networks. By preprocessing the original human body signal of the radar, a micro-Doppler spectrogram representing the micro-Doppler characteristics of human behavior is generated, 1DCNN is used to extract the spatial features of the time series and speed up the network operation efficiency, LSTM is used to capture the time correlation features between sequences to improve the recognition accuracy, the weight allocation mechanism of Attention can focus on more important time features, ensure the effective use of all time and alleviate the problem that the network recognition accuracy decreases due to the long-term memory of LSTM and further improve the recognition accuracy, and the extracted MDF is input into the Softmax classifier to complete the recognition of human behavior. The present invention has the advantages of simple network structure, low computational complexity, fast reasoning speed, high recognition accuracy, etc., and can meet the requirements of accurate and real-time human behavior recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of human behavior recognition, and in particular to a human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network. Background Art

[0002] Human action recognition (HAR) has broad application prospects in many fields such as disaster relief, medical monitoring, smart home, human-computer interaction, etc. These fields require rapid and accurate judgment of human behavior. At present, sensors used for HAR are mainly divided into wearable and non-wearable types. Wearable devices include accelerometers, gyroscopes, etc. Wearable devices require users to wear different sensors on their limbs to collect data. This method has good recognition effect, but the disadvantage is that it causes physical discomfort and has limited usage scenarios. Non-wearable devices include cameras, radars, etc. Camera-based HAR has privacy leakage problems and is not suitable for dark or other low-light scenes. On the contrary, radar-based HAR is more prominent due to its advantages such as all-day and all-weather, strong privacy, and adaptability to any lighting conditions.

[0003] In HAR based on radar micro-Doppler characteristics, the Doppler generated by human trunk movement provides little feature difference, while the features with higher discrimination mainly come from the micro-Doppler generated by the swing of limbs. The micro-Doppler feature (MDF) can clearly reflect the law of human movement and is the basis for identifying human behavior. The original human echo signal collected by radar is a time series data, which may contain background clutter or invalid motion areas. Direct use of this data will result in a large amount of signal processing and is not conducive to the effective extraction of MDF. Based on the radar micro-Doppler characteristics, combining deep learning (DL) networks to extract MDF to achieve HAR is a hot topic in current research. Compared with traditional machine learning, the advantages of DL networks are that they do not need to rely on prior knowledge to manually extract features, have excellent performance, and have strong generalization ability. Typical DL network CNN and its various deep variants have been widely used in HAR tasks. However, most CNNs only extract spatial features and ignore the temporal dependency between time series, which limits the recognition accuracy. In addition, the deeper the network layers, the larger the number of parameters and the higher the hardware requirements.

[0004] Different networks for timing problems (RNN, LSTM, Bi-LSTM, etc.) have made some research progress in HAR. In order to make up for the shortcomings of a single timing network, mixing multiple different networks has become a more common method. For example, the combined structure of CNN-LSTM can fully extract the spatial-temporal features of the timing and obtain better recognition accuracy. In addition, in real-time HAR tasks, it is necessary to respond quickly to human behavior, which requires the network to have an efficient reasoning speed. However, the above-mentioned methods do not fully consider or sacrifice the network operation efficiency to improve the recognition accuracy, which is not conducive to the real-time performance of HAR. Therefore, the processing of radar human micro-Doppler signals and the reasonable design of hybrid multi-networks determine the final recognition effect of HAR. Improving recognition accuracy while ensuring real-time performance is an important research direction in HAR tasks. Summary of the invention

[0005] The purpose of the present invention is to provide a human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network, combining the micro-Doppler characteristics in frequency modulated continuous wave radar and CLA hybrid multi-network, based on an efficient HAR method, to solve the above problems.

[0006] To achieve the above purpose, the present invention provides the following technical solution:

[0007] The human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network includes the following steps:

[0008] S1: The collected radar human body echo signal is preprocessed to generate a multi-frame range-Doppler map (RDM). The preprocessing process includes 2DFFT, background clutter suppression, and constant false alarm rate (CFAR) detection.

[0009] S2: N F The two-dimensional RDM preprocessed by S1 is accumulated in multiple frames under the frame. In this way, the three-dimensional radar data of the distance and speed of each scattering point of the human body changing with the time frame can be obtained. This three-dimensional data is projected on the speed-time dimension, and finally accumulated frame by frame to generate MDM.

[0010] S3: Use CLA (CNN-LSTM-Attention) hybrid multi-network to extract MDF features from the MDM dataset constructed in S2 to complete human behavior recognition.

[0011] Preferably, in step S1, the radar receives the signal reflected by the human body and the surrounding environment, and mixes it with the transmitted signal, and the output result is an intermediate frequency signal.

[0012] Preferably, the RDM of the target can be obtained by using 2DFFT in step S1, and the RDM can intuitively observe the distance and speed related information of all human body scattering points in the corresponding frame.

[0013] Preferably, in step S1, in order to prevent the human body motion signal from being masked by a clutter environment with strong interference such as the background, a notched band-stop filter is placed at zero Doppler to remove most of the strong stationary clutter.

[0014] Preferably, in step S1, in order to reduce the targetless area in the RDM and save signal processing time, a moving average-ordered statistics constant false alarm rate (MAOS-CFAR) algorithm is used to lock the human body motion area.

[0015] Preferably, in step S3, the MDM is divided into many sequences of the same time step in chronological order, with no overlap between each time sequence, and a 1DCNN expansion channel with three convolutional layers is used to obtain multi-dimensional spatial information of the time sequence.

[0016] Preferably, in step S3, in order to characterize the contextual correlation information of the sequence, the long-term memory of LSTM is used to extract the temporal correlation features between the sequences.

[0017] Preferably, in step S3, in order to alleviate the problem of decreased network recognition accuracy caused by long-term memory of LSTM, an Attention weight allocation mechanism is introduced to focus on more important time features from the hidden state of LSTM.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. The human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network provided by the present invention combines the micro-Doppler characteristics in frequency modulated continuous wave radar and CLA hybrid multi-network, and completes the recognition of human behavior based on the efficient HAR method. The present invention has the advantages of simple network structure, low computational complexity, fast reasoning speed, high recognition accuracy, etc., and can meet the requirements of accurate and real-time human behavior recognition.

[0020] 2. This human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network performs a series of signal processing on the original radar echo data to generate a clearer human behavior MDM, which is conducive to extracting MDF from it and improving the recognition accuracy of human behavior.

[0021] 3. This human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-networks takes advantage of the respective advantages of multiple networks to fully express the MDF in MDM, divides MDM into sequences with the same time step, and extracts the spatial-temporal features of the time series through 1DCNN-LSTM. Using 1DCNN to process the time series can speed up the network operation efficiency. LSTM can capture the time correlation features between sequences to achieve long-term dependence. Attention can focus on more important time features to ensure the effective use of all time, alleviate the problem of LSTM's long-term memory leading to a decrease in network recognition accuracy, and improve the network's reasoning speed and recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0023] Figure 1 It is a schematic diagram of the overall process of the human behavior recognition method according to an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of extracting MDF from a CLA hybrid multi-network according to an embodiment of the present invention;

[0025] Figure 3 This is a result diagram before RDM clutter removal according to an embodiment of the present invention;

[0026] Figure 4 This is a result diagram after RDM clutter removal according to an embodiment of the present invention;

[0027] Figure 5 A micro-Doppler spectrum of human walking behavior according to an embodiment of the present invention;

[0028] Figure 6 A micro-Doppler spectrum of a human body running behavior according to an embodiment of the present invention;

[0029] Figure 7 A micro-Doppler spectrum of a human body squatting and standing up behavior according to an embodiment of the present invention;

[0030] Figure 8 A micro-Doppler spectrum of a human body bending behavior according to an embodiment of the present invention;

[0031] Fig. 9 It is a micro-Doppler spectrum of a human turning behavior according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] See also Figure 1-9 The human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network provided by the embodiment of the present invention comprises the following steps:

[0034] S1: The collected radar human body echo signal is preprocessed to generate a multi-frame range-Doppler map (RDM). The preprocessing process includes 2DFFT, background clutter suppression, and constant false alarm rate (CFAR) detection.

[0035] S1.1: The radar receives the signal reflected from the human body and the surrounding environment, mixes it with the transmitted signal, and the output result is an intermediate frequency signal. Considering the situation that the radar transmits multiple frames of signals, the intermediate frequency signal can be expressed as:

[0036]

[0037] In the formula, f c is the radar carrier frequency, B is the signal bandwidth, and T is the signal duration. R is the distance between the radar and the target, V is the radial velocity of the target, c is the speed of light, n = 0, 1, 2, ..., N-1 is the number of sampling points on a single chirp, l = 0, 1, 2, ..., L-1 is the number of chirps in a single frame, and t is the current tth frame.

[0038] S1.2: The RDM of the target can be obtained by using 2DFFT. RDM can intuitively observe the distance and speed related information of all human scattering points in the corresponding frame. The output matrix of (1) after 2DFFT is expressed as:

[0039]

[0040] S(r,v,t) represents the r in RDM in frame t th Range gate, v th The signal power value under the speed gate, RDM, can be converted from S(r,v,t):

[0041] RD(r,v,t)=|S(r,v,t)|#(3)

[0042] S1.3: In order to prevent the human motion signal from being masked by the background and other strong interference clutter environment, a notched band-stop filter is placed at zero Doppler to remove most of the strong stationary clutter without affecting the human motion signal. The two-dimensional matrix after clutter removal is

[0043] S1.4: In order to reduce the targetless area in RDM and save signal processing time, a sliding average-ordered statistics constant false alarm rate (MAOS-CFAR) algorithm is used to lock the human motion area. The algorithm uses a window of length U+Y in Slide in the middle, where U is the total length of the reference unit, Y is the detection unit, sort the values ​​of the reference units on both sides of Y from small to large, and then select the kth value to estimate Z k , k∈[3U / 4,5U / 4], the threshold factor α is determined by the following formula:

[0044]

[0045] Where P fa is the false alarm probability. If Y is greater than the detection threshold (θ = α × Z k ), it is determined that there is a human motion signal in the corresponding sliding window, and the range gate number r in this sliding window is saved. min and r max After the sliding window is completed, a two-dimensional matrix containing the human body motion range can be obtained Finally, the RDM of the human motion area is generated according to formula (3).

[0046] S2: N F The two-dimensional RDM preprocessed by S1 is accumulated in multiple frames under the frame. In this way, the three-dimensional radar data of the distance and speed of each scattering point of the human body changing with the time frame can be obtained. This three-dimensional data is projected on the speed-time dimension, and finally accumulated frame by frame to generate MDM. The total accumulation time is N T , the time interval of each MDM frame is N T / N F , using the generated MDM to construct a human behavior dataset, including the training dataset O train And the test data set O test .

[0047] S3: Use CLA hybrid multi-network to extract MDF features from the MDM dataset constructed in S2 to achieve HAR. The whole process is as follows Figure 2 shown.

[0048] S3.1: Divide the MDM into many tiles of the same time step in chronological order, with no overlap between tiles. Each tile can be regarded as a one-dimensional micro-Doppler time series. Vectorize these time series to obtain the 1DCNN input sequence at the corresponding time step: M = (m 1 ,m 2 ,…,m T ), after all time steps, the input of the entire MDM time series is completed.

[0049] S3.2: Use 1DCNN with three convolutional layers to expand the channel to obtain multi-dimensional spatial information of the time series. The size of each convolution kernel is C, the number of convolution kernels is N, and the local spatial features of the t-th input sequence are extracted by continuously sliding the convolution kernel. 1DCNN outputs a multi-channel feature map, which is converted into a feature sequence X=(x 1 ,x 2 ,…,x T ).

[0050] S3.3: The feature sequence X generated in S3.2 is converted into (x 1 ,x 2 ,…,x T ) is input into LSTM to extract the temporal correlation features between sequences. At time step t, the input sequence of LSTM unit is x t , and get the corresponding forget gate f t , input gate i t , output gate o t 、Memory state c t , hidden state h t , respectively as follows:

[0051]

[0052] In the formula, sigmoid and tanh are activation functions, W (·) is the weight matrix, [x t ,h t-1 ] is the input x at time t t With the previous hidden state h t-1 When all X are output through LSTM, the hidden state H = (h 1 ,h 2 ,…,h T ), and the hidden state h of the last time step output can also be obtained T .

[0053] S3.4: In order to alleviate the problem that long-term LSTM memory leads to network redundancy and reduced recognition accuracy, the Attention mechanism can evaluate which time steps in the LSTM output H generate hidden states that are important for h T The biggest impact is to convert the hidden state h output by LSTM at each time step i (i=1,…,T)) and h T Calculate the similarity and get a score:

[0054] score(h T ,h i )=h T T h i #(6)

[0055] Use softmax to convert this score into Attention weight:

[0056]

[0057] For each h i Each of them will learn an Attention weight. The larger the weight, the more h i In h T China is more important.

[0058] α T,i With each h i Perform weighted summation to obtain the weight factor:

[0059]

[0060] A T With h T Concatenate to get the Attention hidden state:

[0061] h A =tanh(W c [A T ;h T ])#(9)

[0062] The newly calculated Attention hidden state h A Compared to h T Contains more important time features, h A Connect the Softmax layer to convert it into class conditional probability:

[0063] p=softmax(W h h A +b h )#(10)

[0064] Where W his the weight matrix, b h For bias.

[0065] S3.5: O train All training samples of CLA network are trained according to steps 3.1-3.4, and the trained network is used for O test Sample data, testing the final recognition effect of HAR.

[0066] The practical method provided by the present invention is applied to actual HAR to verify the performance of the method. The specific implementation steps are as follows:

[0067] The experiment uses TI AWR1843 radar to collect human behavior data. The radar parameters are set as follows: carrier frequency 77GHz, sawtooth frequency modulation continuous wave, sampling points 128, Doppler resolution 0.05m / s, single behavior acquisition time 5s. In order to achieve a sufficient detection area, the radar is set up at a height of 1.5m, and the experimenter is 3-4m away from the radar. The experiment designed five common human behaviors, namely walking, running, squatting and standing up, bending over and turning around. In order to make the data generalizable, the experiment selected 7 men and 3 women to participate in data collection. Their height, weight and age are different. 10 experimenters repeated the experiment. Each behavior was performed 20 times, for a total of 1000 (5×10×20) sets of human behavior data. When collecting experimental data, only a single experimenter scenario was used. The behavior was not strictly constrained and could be performed according to personal habits to ensure the diversity of the data set. The radar raw data processing process was implemented in MATLAB v2017. The output size of 2DFFT was 134×256. The original RDM contained strong stationary clutter. If it was not removed, it would affect the subsequent MDM generation. The RDM results before and after clutter removal are shown in Figure 2. Figure 3 and Figure 4 As shown in Figure 1, 50ms produces one frame of MDM, and the total accumulation time of 100 frames of MDM is 5s. The MDM of five types of human behaviors is as follows: Figure 5-Figure 9 As shown in the figure, different human behaviors have unique MDFs, which are the basis for human behavior recognition. The MDM is converted into a grayscale image and the image size is reshaped to 112×112. The original details of the MDM are not lost after the reshaping, and the calculation amount of the CLA hybrid multi-network can be reduced. In order to ensure that there are enough samples in the data set, data enhancement is also used. The total amount of data generated is 4950, which is divided into O and B according to the ratio of 7:3. train and O test(3465:1485), the CLA network was trained in the DL framework of Tensorflowv1.4.0 and python3.6, and the network parameters were set as follows: time step 112, MDM was divided into tiles with the same time step, each tile size was 1×112, and the input of MDM was completed after all time steps, 3 1DCNN convolution layers, 3 convolution kernels per layer, and the number of convolution kernels was 16, 32, and 64 respectively. The number of LSTM layers was 1, the number of hidden neurons was 32, and the Adam optimizer was used for training the network, the learning rate, the number of training rounds, and the number of batch samples for each iteration were 0.001, 150, and 690 respectively. At the same time, the experiment used the Dropout method to close some neurons in the network to avoid the problem of overfitting leading to reduced generalization ability of the network. The above experiments were all based on an Inteli7-8-core CPU and an Intel UltraCore 630 PC.

[0068] The performance comparison results of different networks are shown in the following table:

[0069] Network Name Average accuracy Parameter quantity Training time Inference time / sample 2DCNN 96.49％ 86021 2589.22s 2.91ms RNN 71.34％ 4805 72.96s 0.28ms LSTM 95.23％ 18725 290.39s 0.93ms Bi-LSTM 98.56％ 37445 630.28s 1.22ms 1DCNN-LSTM 96.74％ 25749 435.32s 0.82ms CLA 99.36％ 56165 523.05s 0.96ms

[0070] This method compares the parameters, training time, single sample inference time and average test accuracy of several common HAR networks under the same network structure. The experimental results are shown in the table above. It can be seen that the average accuracy of this method can reach more than 99%, which is the best among all networks. Although the proposed method requires a long training time, considering the two factors of accuracy and real-time performance, the overall performance of this method is better than that of other networks and can meet the requirements of HAR accuracy and real-time performance.

[0071] The human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network provided by the embodiment of the present invention generates a micro-Doppler spectrogram representing the micro-Doppler characteristics of human behavior by preprocessing the original human body signal of the radar, uses 1DCNN to extract the spatial characteristics of the time series and speed up the network operation efficiency, uses LSTM to capture the time correlation characteristics between sequences to improve the recognition accuracy, and the weight allocation mechanism of Attention can focus on more important time features, ensure the effective use of all time, and alleviate the problem of the decline in network recognition accuracy caused by the long-term memory of LSTM and further improve the recognition accuracy. The extracted MDF is input into the Softmax classifier to complete the recognition of human behavior. The present invention has the advantages of simple network structure, low computational complexity, fast reasoning speed, high recognition accuracy, etc., and can meet the requirements of accurate and real-time human behavior recognition.

[0072] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0073] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network, It is characterized in that The following steps are involved: S1: Preprocess the collected radar human echo signal to generate a multi-frame range-Doppler map (RDM). The preprocessing process includes 2DFFT, background clutter suppression, and constant false alarm rate (CFAR) detection; S2: The two-dimensional RDM preprocessed by S1 is accumulated in multiple frames under the frame. In this way, the three-dimensional radar data of the distance and speed of each scattering point of the human body changing with the time frame can be obtained. This three-dimensional data is projected on the speed-time dimension, and finally accumulated frame by frame to generate MDM; S3: Use CLA (CNN-LSTM-Attention) hybrid multi-network to extract MDF features from the MDM dataset constructed in S2 to complete human behavior recognition; In the step S1, the RDM of the target can be obtained by using 2DFFT. The RDM can intuitively observe the distance and speed related information of all human scattering points in the corresponding frame, which is specifically expressed as: ； Among them, S(r,v,t) represents the RDM in frame t. Distance gate, The signal power value under the speed gate, RDM can be obtained by Convert to get; In step S1, in order to prevent the human motion signal from being covered by the background clutter environment with strong interference, a notched band-stop filter is placed at zero Doppler to remove most of the strong stationary clutter without affecting the human motion signal. The two-dimensional matrix after clutter removal is ; In step S1, in order to reduce the targetless area in RDM and save signal processing time, a sliding average-ordered statistics constant false alarm rate (MAOS-CFAR) algorithm is used to lock the human motion area; the algorithm uses a window with a length of U+Y in Slide in the middle, where U is the total length of the reference unit, Y is the detection unit, sort the values ​​of the reference units on both sides of Y from small to large, and then select the kth value to estimate , k∈[3U / 4,5U / 4], the threshold factor α is determined by the following formula: ； In the formula, is the false alarm probability, if Y is greater than the detection threshold θ=α× , it is determined that there is a human motion signal in the corresponding sliding window, and the range gate number in this sliding window is saved. and After the sliding window is completed, a two-dimensional matrix containing the human body motion range can be obtained , finally according to Generate RDM of human motion regions; In step S3, in order to alleviate the problem of decreased network recognition accuracy caused by long-term memory of LSTM, the Attention weight distribution mechanism is introduced to focus on more important time features from the hidden state of LSTM. The Attention mechanism can evaluate which time steps in the LSTM output H generate the hidden state that is most important to the network recognition accuracy. The biggest impact is to convert the hidden state of the LSTM output at each time step , i=1,…,T and Calculate the similarity and get a score: ； Use softmax to convert this score into Attention weight: ； For each Each time, an Attention weight is learned. The larger the weight, the exist China is more important than Vietnam; Will With each Perform weighted summation to obtain the weight factor: ； Will and Concatenate to get the Attention hidden state: ； In the formula, is the weight matrix; Newly calculated Attention hidden state Compared to Contains more important time features, Connect the Softmax layer to convert it into class conditional probability: ； In the formula, is the weight matrix, For bias.

2. According to claim 1, the human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network, Features: In step S1, the radar receives the signal reflected from the human body and the surrounding environment, and mixes it with the transmitted signal, and the output result is an intermediate frequency signal.

3. According to claim 1, the human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network, Features: In step S1, 2DFFT is used to obtain the RDM of the target, and the RDM can intuitively observe the distance and speed related information of all human body scattering points in the corresponding frame.

4. The human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network according to claim 1, Features: In step S1, in order to prevent the human body motion signal from being masked by the background clutter environment with strong interference, a notched band-stop filter is placed at zero Doppler to remove most of the strong stationary clutter.

5. The human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network according to claim 1, Features: In step S3, the MDM is divided into many sequences of the same time step in chronological order, with no overlap between each time series, and a 1DCNN expansion channel with three convolutional layers is used to obtain multi-dimensional spatial features of the time series.

6. The human behavior recognition method based on radar micro-Doppler characteristics and hybrid multi-network according to claim 1, Features: In step S3, in order to characterize the contextual association information of the sequence, the long-term memory of LSTM is used to extract the temporal association features between the sequences.

Images