A radar human behavior recognition method based on multi-domain feature fusion

Abstract

The application discloses a radar human behavior recognition method based on multi-domain feature fusion, and is applied to the problem that the existing human behavior recognition method based on a radar sensor and a deep learning technology only adopts the features of one domain or only adopts one time-frequency analysis method in a time-frequency domain, thereby causing insufficient expression of human behavior features; three time-frequency analysis methods with different time-frequency resolutions, namely, a short-time Fourier transform, an adaptive optimal kernel time-frequency representation method and a Hann kernel reduction cross-term interference distribution, are selected; three types of frequency spectrum diagrams are obtained; then, the three types of time-frequency spectrum diagrams are combined and used by using a SE Net and a 3DCNN network in the time-frequency domain, so that human behavior features are more fully expressed; in a distance domain, a key feature is extracted by using the SE Net; and the features of two domains are combined, the mutual relationship is found, and the recognition accuracy of human behaviors is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of radar automatic target recognition technology, and specifically relates to a multi-domain feature fusion human behavior recognition technology. Background Technology

[0002] Human behavior recognition is rapidly becoming a key technology in fields such as disaster relief, intelligent security, nursing care, and military operations. Human behavior recognition is primarily achieved through various sensors. Leveraging high-resolution optical equipment and rapidly developing computer vision technology, vision-based human-computer interaction has yielded abundant research results in this field. However, vision-based human behavior recognition is susceptible to lighting conditions, obstruction, and privacy breaches. Radar-based human behavior recognition overcomes these drawbacks. Compared to other commonly used sensors, radar avoids the temperature effects of infrared sensors and the device requirements of wearable devices. It also possesses strong penetrating power and can adapt to various scenarios. Therefore, radar-based human behavior recognition has broad prospects in this field.

[0003] Therefore, radar-based human behavior recognition technology has attracted great attention from research institutions at home and abroad in recent years. To enrich the feature representation of human behavior and lightweight network structures, scholars from the University of Electronic Science and Technology of China proposed a lightweight human behavior recognition method based on multi-domain fusion (P. Chen, et al. A Multi-domain Fusion Human Motion Recognition Method Based on Lightweight Network). This method first uses a parallel feature pyramid network to extract multi-domain features from the time-frequency map and range map, then uses channel-dimensional concatenation (Concat) to fuse the multi-domain features, and finally uses the existing MobileNet V3 lightweight network to further extract deep features from the fused multi-domain features and obtain the recognition results of human behavior behind a wall. Scholars from the University of Chinese Academy of Sciences (Y. Yao, et al. Fall Detection System Using Millimeter-Wave Radar Based on Neural Network and InformationFusion) designed a robust fall detection system based on frequency modulated continuous wave (FMCW) radar. This system detects the instant of human movement, calculates the range-velocity map, range-horizontal angle map, and range-vertical angle map of the radar signal, and creates three neural networks for these three signal maps. This paper utilizes a stacking method of ensemble learning to fuse spatiotemporal-velocity features extracted from three neural networks to identify falls. However, this method only uses one time-frequency analysis method to obtain the feature spectrum in the time-frequency domain. Furthermore, in terms of multi-domain feature fusion, this method merely concatenates features from multiple domains without considering the correlations between features from different domains. Therefore, researching a human behavior method that fully represents time-frequency features using multiple time-frequency analysis methods and simultaneously utilizes correlated features from multiple domains is of significant research importance. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a radar human behavior recognition method based on multi-domain feature fusion, which combines features from multiple domains to fully characterize human behavior and improve the probability of recognizing human behavior.

[0005] The technical solution adopted in this invention is: a radar human behavior recognition method based on multi-domain feature fusion, comprising:

[0006] S1: Collect human behavior data using linear frequency modulated continuous wave radar;

[0007] S2: Perform an M-point FFT transformation on the fast time column of the collected two-dimensional raw data to obtain a distance spectrum with dimension M×N;

[0008] Find the point with the largest value in each column of the distance spectrum and assemble them into a one-dimensional time series in slow time. Then, perform short-time Fourier transform, adaptive optimal kernel time-frequency representation, and Hanning kernel to reduce the cross-term interference distribution on this one-dimensional time series to obtain three types of time spectrum maps.

[0009] S3: Input the distance spectrum map and the three types of time spectrum maps into the single-domain feature extraction module based on SE-ResNet to obtain the distance spectrum map features and the three types of time spectrum map features;

[0010] S4: Input the three types of time-spectrum features obtained in step S3 into the 3DCNN to obtain the fused multi-resolution time-spectrum features;

[0011] S5: Use Bilinear pooling to fuse the distance domain features processed in step S3 and the multi-resolution time-spectral map features processed in step S4 to obtain a multi-domain feature vector;

[0012] S6: Multi-domain feature vectors are sequentially passed through a 2DCNN and a fully connected layer to complete human behavior classification and obtain recognition results.

[0013] The single-domain feature extraction module consists of a 3×3 convolutional layer followed by ReLU non-linear activation, a 4-layer SE-ResNet module, Avgpool for dimensionality reduction, and finally a fully connected layer to obtain the result.

[0014] The SE-ResNet module consists of a ResNet module nested within an SENet module. Specifically, it includes two convolutional layers with a kernel size of 3×3, followed by an SENet structure whose input and output are connected via a short-circuit structure.

[0015] SENet module: For a C×W×H feature map, the feature map is first subjected to global average pooling through a Squeeze operation to obtain a 1×1×C feature vector. Then, two FC layers form a Bottleneck structure to model the correlation between channels: (1) After the first FC layer, the C channels are transformed into C / r to reduce the number of parameters, and then the second FC layer is reached through ReLU nonlinear activation. (2) After the second FC layer, the number of feature channels is restored to C, and weight parameters with attention mechanism are obtained. Finally, the Sigmoid activation function is passed, and the normalized weights are weighted onto the features of each channel through a Scale operation.

[0016] The beneficial effects of this invention: Based on multi-resolution time-frequency spectrograms and range spectrograms, this invention proposes a radar human behavior recognition method based on multi-domain feature fusion. This method overcomes the low recognition accuracy problems caused by current human behavior recognition methods that only use single-domain feature representation and a single time-frequency analysis method in the time-frequency domain. In the time-frequency domain, this method utilizes an attention mechanism to extract key features from three spectrograms with different time-frequency resolutions, generated by Short-Time Fourier Transform (STFT), Adaptive Optimal Kernel Time-Frequency Representation (AOKTFR), and Hanning Kernel Reduced Cross-Term Interference Distribution (RIDHK). A 3DCNN is used to extract the intrinsic correlation features of the three spectrograms. In the range domain, the attention mechanism is also used to extract key features and reduce redundancy. Then, bilinear pooling is used to fuse the features from the two different domains, thereby comprehensively describing the human behavior characteristics. Finally, a 2DCNN is used to extract the fused multi-domain features to obtain the final classification result. Attached Figure Description

[0017] Figure 1 This outlines the overall framework of the invention method.

[0018] Figure 2 The data processing flow for human behavior radar;

[0019] Figure 3 Here are schematic diagrams of the three types of time-frequency spectra and distance spectra generated;

[0020] Figure 4 This is a schematic diagram of the spectral feature extraction module based on SE-ResNet;

[0021] Figure 5 This is a schematic diagram of the constructed 2DCNN network and fully connected layers. Detailed Implementation

[0022] To facilitate understanding of the technical content of this invention by those skilled in the art, the following description, in conjunction with the accompanying drawings, further illustrates the invention.

[0023] like Figure 1 As shown, the method implementation process of the present invention includes the following steps:

[0024] S1: Collecting human behavior data via linear frequency modulated continuous wave radar:

[0025] (1) The center frequency of the linear frequency modulated continuous wave radar is 60 GHz.

[0026] (2) The effective bandwidth of the linear frequency modulated continuous wave radar is 960MHz.

[0027] (3) The period of the linear frequency modulated continuous wave radar Chrip (linear frequency modulation) signal is 50us.

[0028] After collection, the data is preprocessed according to the following steps:

[0029] 11: Verify the integrity of human behavior data: The linear frequency modulated continuous wave radar transmits 64 Chrip signals per frame, and each Chrip signal samples 128 complex signals. One human behavior dataset is collected for 100 frames, totaling 5 seconds of radar data. After reading the data, each set of experimental data should have a complex vector of length 819200. If the data length does not meet the requirement, the set of data should be discarded.

[0030] 12: Rearrange the human behavior data: Extract each Chrip signal from a complex vector of length 819200 and rearrange the data. The rearranged data is a two-dimensional original data matrix of 128×6400.

[0031] S2: Use signal processing methods to obtain the range spectrum and three types of time spectrum.

[0032] Its signal processing flow is as follows Figure 2 As shown. The acquired two-dimensional raw data matrix undergoes a 256-point FFT (Fast Fourier Transform) transformation on the fast time column to obtain a distance spectrum with dimensions of 256×6400. The point with the largest value is found in each column of the distance spectrum, and these points are concatenated into a one-dimensional vector s on the slow time column. For this one-dimensional time series s, the Short-Time Fourier Transform (STFT) is defined as:

[0033]

[0034] Where t is time, f is frequency, τ is the convolution variable, s is the information data vector, h(·) is the frequency window function, and h * (·) represents the conjugate of the data. Here, the window length is chosen to be 17. The Adaptive Optimal Kernel Time-Frequency Distribution (AOKTFR) is defined as:

[0035] AOKTFR(t,f)=∫∫AF(θ,τ)Φ opt (θ,τ;t,ω,s)×e -jθt-jτω dθdτ

[0036] Where AF(·) represents the fuzzy function, Φ opt (·) represents the optimal kernel, s is the information data vector, t is time, f is frequency, ω = 2πf, τ is the convolution variable, and θ is the angle on the fuzzy plane. The Hanning kernel for reducing cross-term interference distribution is defined as:

[0037]

[0038]

[0039] Where t is time, f is frequency, τ is the convolution variable, s is the information data vector, h(·) represents the frequency window function, and g(·) represents the time window function. Here, the frequency window length is chosen to be 17, and the time window length is chosen to be 18. The resulting three types of time-spectral maps and distance spectra are as follows: Figure 3 As shown. From Figure 3 As can be seen, the spectrogram generated by Short Time Fourier Transform (STFT) has the lowest time-frequency resolution, but also introduces some redundant features; the spectrogram generated by Hanning Kernel Reduced Cross-Term Interference Distribution (RIDHK) has a higher time-frequency resolution, but is affected by cross-term interference; the spectrogram generated by Adaptive Optimal Kernel Time-Frequency Representation (AOKTFR) has a high resolution and is less affected by cross-term interference. However, it expresses fewer features. These three spectrograms contain different relevant information. The distance spectrogram, on the other hand, has completely different distance characteristics.

[0040] Those skilled in the art should note that the number of categories in the time-spectrum graph of this invention is not limited to three, but there will be limitations if there are only one or two categories; the more categories there are, the more features there are, but the complexity will increase and the recognition effect may be worse; in this embodiment, three categories are used, and 3DCNN is used to extract the association between different categories.

[0041] S3: Extract features for each class of spectra using a single-domain feature extraction module based on SE-ResNet.

[0042] The SE-ResNet module network structure is as follows: Figure 4 As shown in (a), first there are two convolutional layers with a kernel size of 3×3, followed by an SENet structure, and its input and output are connected through a short-circuit structure. The working principle of the SENet module is as follows: For a C×W×H feature map, the feature map is first subjected to global average pooling through the Squeeze operation to obtain a 1×1×C feature vector. Then, two FC layers form a Bottleneck structure to model the correlation between channels: (1) After the first FC layer, the C channels are transformed into C / r to reduce the number of parameters, and then the non-linear activation of ReLU is used to reach the second FC layer. (2) After the second FC layer, the number of feature channels is restored to C, and weight parameters with attention mechanism are obtained. Finally, after the Sigmoid activation function, the normalized weights are weighted onto the features of each channel through a Scale operation.

[0043] For an input spectrum with dimensions of 224×224×3, the feature extraction process is as follows: Figure 4 As shown in (b): First, a 3×3 convolutional layer (Conv) with 64 kernels is passed, resulting in a feature vector size of 64×224×224. Then, a first SE-ResNet layer with 64 kernels is passed, resulting in a size of 64×224×224; a second SE-ResNet layer with 128 kernels is passed, resulting in a size of 128×112×112; a third SE-ResNet layer with 256 kernels is passed, resulting in a size of 256×56×56; a fourth SE-ResNet layer with 256 kernels is passed, resulting in a size of 256×28×28; finally, after pooling (Avgpool) and fully connected (FC), the feature vector size becomes 256×1×1. Those skilled in the art should understand... Figure 4 BN (Batch Normalization) in the text is the normalization layer.

[0044] S4: Use 3DCNN to combine the features of the three types of time spectrograms to obtain the fused multi-resolution time spectrogram features.

[0045] The feature maps of the three types of time-frequency spectrograms each have a dimension of 256×1. These are stacked, resulting in a size of 256×1×1×3. They are then fed into a first layer with 256 convolutional kernels of size 3×3×3 for convolution, resulting in a size of 256×1×1×3. Next, they are fed into a second layer with 512 convolutional kernels of size 3×3×3 for convolution, resulting in a size of 512×1×1×3. After pooling and a fully connected layer, the size becomes 512×1. This yields a time-frequency domain feature vector with multi-resolution time-frequency features.

[0046] S5: Use Bilinear pooling to fuse distance domain features and time-frequency domain features.

[0047] At this point, the feature vector dimension in the distance domain is 256×1, and the feature vector dimension in the time-frequency domain is 512×1.

[0048] b(ι,χ,f A ,f B )=f A ×f B

[0049]

[0050] x=vec(ξ(χ))

[0051]

[0052] z = y / ||y||2

[0053] ||·||2 represents the 2-norm.

[0054] The feature map χ at position ι has a feature vector of dimension 256×1 in the distance domain, and its feature vector is f. A The eigenvector with dimension 512×1 in the time-frequency domain is f. B The bilinear multiplication of the two features yields matrix b. The sum of all values ​​of b at all positions results in matrix ξ(χ). Finally, matrix ξ(χ) is truncated into a vector using the vec() operation. After performing the normalization sign() operation, the fused feature z is obtained, thus completing the feature fusion between the two domains. A multi-domain feature vector with dimension 4096×1 is obtained.

[0055] S6: Use 2DCNN and fully connected layers to extract features after multi-domain fusion to complete human behavior classification.

[0056] 2DCNN and fully connected network architectures such as Figure 5 As shown, for a multi-domain feature vector with an input of 4096×1, the size is reduced to 2048×1 after passing through a first convolutional layer (Conv) with 2048 3×3 2D convolutional kernels; the size is reduced to 1024×1 after passing through a second convolutional layer (Conv) with 1024 3×3 2D convolutional kernels; the size is reduced to 512×1 after passing through a third convolutional layer (Conv) with 512 3×3 2D convolutional kernels; the size is reduced to 256×1 after passing through a fourth convolutional layer (Conv) with 256 3×3 2D convolutional kernels; the 256×1 tensor is flattened into a vector using Flatten; finally, a fully connected layer (FC layer) is passed to obtain a classification result probability of size 6×1.

[0057] The effectiveness of this invention is further illustrated by the following experiments:

[0058] Training and test set construction:

[0059] Six human behaviors were collected using radar: waving, falling, sleeping, getting up, standing, and sitting. Data was collected from 10 participants of different genders, heights, and weights. Each behavior was repeated 60 times per participant, resulting in 3600 sets of experimental data. 2520 sets were randomly selected as the training set, and the remainder were used as the test set.

[0060] Network training:

[0061] The experiments were conducted on the PyTorch machine learning platform using an NVIDIA GeForce GTX 3080 Ti graphics card. The learning rate of the models was set to 0.01. Each model was trained for 300 epochs.

[0062] Network testing:

[0063] To verify the effectiveness of the proposed method, we extracted the network architectures used in the ablation experiments from the proposed method and then compared them. These included the SE-ResNet-3DCNN network based on multi-resolution spectrograms and the SE-ResNets network based on single-class spectrograms.

[0064] The network was tested using test set samples and compared with the aforementioned networks. The recognition probabilities of each method are shown in Table 1.

[0065] Table 1 Comparison of recognition probabilities between the method of the present invention and existing technologies

[0066]

[0067] Experiments show that the present invention can effectively identify human behavior, and its performance is superior to that of classic human behavior recognition methods, thus verifying the effectiveness of the present invention.

[0068] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A radar-based human behavior recognition method based on multi-domain feature fusion, characterized in that, include: S1: Collect human behavior data using linear frequency modulated continuous wave radar; S2: Perform an M-point FFT transformation on the fast time column of the collected two-dimensional raw data to obtain the dimension... Distance spectrum; Find the point with the largest value in each column of the distance spectrum and assemble them into a one-dimensional time series in slow time. Use various time-frequency analysis methods to characterize the time-frequency features of this one-dimensional time series to obtain multiple types of time spectrum maps. The various time-frequency analysis methods described in step S2 characterize time-frequency features, specifically including the following three time-frequency analysis methods: Short-time Fourier transform, adaptive optimal kernel time-frequency representation, and Hanning kernel to reduce cross-term interference distribution; S3: Input the distance spectrum and multi-class time spectrum into the single-domain feature extraction module based on SE-ResNet to obtain the distance spectrum features and multi-class time spectrum features; S4: Fuse the multi-class time-spectrum features obtained in step S3 to obtain fused multi-resolution time-spectrum features; S5: Use bilinear pooling to fuse the distance domain features processed in step S3 and the multi-resolution time-spectral map features processed in step S4 to obtain a multi-domain feature vector. S6: Classify and recognize human behavior based on multi-domain feature vectors.

2. The radar human behavior recognition method based on multi-domain feature fusion according to claim 1, characterized in that, The single-domain feature extraction module based on SE-ResNet described in step S3 specifically includes: a convolutional layer with a kernel size of 3×3, a ReLU nonlinear activation layer, a 4-layer SE-ResNet module, an average pooling layer, and a fully connected layer. The SE-ResNet module specifically includes: two convolutional layers with a kernel size of 3×3, followed by an SENet structure whose input and output are connected through a short-circuit structure; The SENet module specifically includes: a compression layer, two fully connected layers, a sigmoid activation function layer, and a normalization layer.

3. The radar human behavior recognition method based on multi-domain feature fusion according to claim 2, characterized in that, Step S4 uses 3DCNN to fuse the three types of temporal spectrogram features.

4. The radar human behavior recognition method based on multi-domain feature fusion according to claim 3, characterized in that, Step S6 specifically involves processing the multi-domain feature vectors sequentially through a 2DCNN and a fully connected layer to complete the human behavior classification and obtain the recognition results.

Images