Multi-human-body-target behavior recognition method based on millimeter-wave radar

Abstract

The invention discloses a multi-human-body-target behavior recognition method based on millimeter-wave radar. The method comprises the following steps: firstly, acquiring human body behavior data by deploying a millimeter wave radar, and obtaining a multi-dimensional parameter signal of human body behaviors by adopting 4D-Fast Fourier Transform (4D-FFT); secondly, a high-dimensional matrix signal is converted into a point cloud matrix through a cell average constant false alarm rate (CAFAR) detection algorithm (CA-CFAR) and geometric space coordinate conversion, so that the complexity of a radar signal is effectively reduced; secondly, further compressing the point cloud matrix through a density-based spatial clustering (DBSCAN) algorithm and a local outlier factor (LOF) algorithm based on space and speed information to remove dynamic interference and static interference in the environment, and then realizing segmentation of the multi-target point cloud matrix through a Hungary algorithm to obtain a pure single-target point cloud matrix; and finally, realizing accurate recognition of human body behaviors through a neural network based on the multi-dimensional radar point cloud. According to the invention, through radar point cloud and deep learning, the complexity of radar signals is reduced and accurate recognition of human body behaviors is realized at the same time.

Description

Technical Field

[0001] The present invention belongs to human behavior recognition technology, and in particular relates to a multi-human target behavior recognition method based on millimeter wave radar. Background Art

[0002] With the continuous evolution of computer vision and sensor technology, human body recognition technology is increasingly being used in various fields. This technology not only plays a key role in intelligent monitoring, security systems, and intelligent transportation, but the widespread application of human body recognition technology will profoundly change people's lifestyles and working methods, and promote the continuous development of society towards a more intelligent and convenient direction.

[0003] There are two main types of traditional human behavior recognition methods, namely wearable device-based behavior recognition and vision-based behavior recognition. In wearable device-based behavior recognition, users wear electronic devices with integrated data sensors, measure behavior data through sensors such as speedometers or gyroscopes in the device, and complete the recognition task based on the characteristics of the measured data. However, wearable devices are difficult to popularize because they bring inconvenience to users. Another type of vision-based method uses images or video data captured by cameras to achieve recognition tasks through traditional image processing methods or machine learning technologies. This method can achieve high recognition accuracy at a low cost, and the technology is relatively mature. However, the performance of cameras will drop sharply in harsh environments such as high ambient light ratio, severe rain, snow, flying dust and sand, or target occlusion, which limits the application of this method in complex environments. In addition, in modern society, visual images or videos also pose great challenges to user privacy.

[0004] In contrast, it has become a hot research field to obtain target behavior information based on non-contact sensors to complete behavior recognition and classification tasks, such as sound, infrared and radar sensor devices. Among these devices, radar, as one of the main sensors, has the advantages of not being affected by light and being able to penetrate obstacles. On the other hand, radar signals can provide information such as the distance, speed and angle of the target, making it more sensitive to the target's motion information and suitable for target detection and tracking in complex environments. In recent years, some studies have begun to explore the use of deep learning technology to learn features and classify targets for radar signals, thereby improving the accuracy and applicability of human recognition.

[0005] Millimeter wave radar, with its high resolution, high precision, privacy and non-contact characteristics, can collect target behavior information around the clock, and has great potential and advantages in the field of target detection. The present invention discloses a method for multi-human target behavior recognition based on millimeter wave radar. Firstly, by deploying millimeter wave radar to collect human behavior data, 4D-Fast Fourier transform (4D-FFT) is used to obtain multi-dimensional parameter signals of human behavior; secondly, the high-dimensional matrix signal is converted into a point cloud matrix through the Cell Averaging Constant False-Alarm Rate (CA-CFAR) and geometric space coordinate transformation to effectively reduce the complexity of radar signals; then, the point cloud matrix is ​​further compressed through the density clustering (DBSCAN) and local outlier factor (LOF) algorithm based on spatial and velocity information to remove dynamic and static interference in the environment, and then the Hungarian algorithm is used to segment the multi-target point cloud matrix to obtain a pure single-target point cloud matrix; finally, the accurate recognition of human behavior is achieved through a neural network based on multi-dimensional radar point cloud. The patent of this invention combines radar point cloud with deep learning technology to reduce the complexity of radar signals while achieving accurate recognition of human behavior. Summary of the invention

[0006] The purpose of the present invention is to provide an accurate and lightweight method for multi-human target recognition, which captures the motion information of multiple human targets through millimeter wave radar, extracts the pure single human target motion point cloud matrix from the radar signal through a series of signal processing algorithms, and completes the behavior recognition task through a dedicated point cloud neural network.

[0007] The invention discloses a method for identifying multiple human target behaviors based on millimeter wave radar, comprising the following steps:

[0008] Step 1: Use frequency modulated continuous wave (FMCW) radar to collect radar signals of multiple human target behaviors. By performing four FFT transformations on the fast time, slow time, azimuth antenna and elevation antenna of the echo signal matrix, we can obtain an echo signal matrix with four dimensions of information including distance, speed, azimuth and elevation.

[0009] Step 2: The present invention eliminates virtual power through the CA-CFAR algorithm, performs preliminary target detection, and retains signal units that may contain targets. Secondly, the high-dimensional matrix signal is converted into a point cloud matrix through geometric space coordinate transformation, effectively reducing the complexity of the radar signal.

[0010] Step 3: Use the DBSCAN algorithm to cluster the point cloud matrix in one step. Set the neighborhood distance threshold ε and the sample number threshold MinPts based on the space and speed information, group the qualified point clouds into one cluster, and mark the isolated point clouds that do not belong to any cluster as noise and remove them. Secondly, use the local outlier factor algorithm to calculate the outlier factor LOF of each point, set an appropriate threshold, and filter out the outliers of the point cloud matrix more carefully to remove the interference signals and thermal noise in the environment. Thirdly, use the Hungarian algorithm to track the target frame by frame according to the time frame sequence of the radar, match each point cloud in turn, realize the segmentation of the multi-target point cloud matrix, and obtain a pure single-target point cloud matrix.

[0011] Step 4: Build a point cloud neural network based on multi-dimensional FMCW point cloud for action recognition tasks. Input the pure single-target point cloud matrix obtained in step 3 into the neural network for training, adjust the network parameters, and perform recognition after the neural network converges. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of the present invention;

[0013] Figure 2 This is a visualization of the point cloud matrix after CA-CFAR and coordinate transformation.

[0014] Figure 3 This is a visualization of the pure single-target point cloud matrix after step processing.

[0015] Figure 4 This is the basic structure diagram of the point cloud neural network. Specific implementation plan

[0016] The present invention will be further described below in conjunction with the accompanying drawings:

[0017] like Figure 1 The method for multi-human target behavior recognition based on millimeter wave radar shown in the figure specifically comprises the following steps:

[0018] Step 1: Use frequency modulated continuous wave radar to collect radar signals of multiple human target behaviors. The radar echo signal is mixed with the transmission signal to obtain an intermediate frequency signal containing target information. By performing four FFT transformations on the fast time (i.e. sampling point dimension), slow time (i.e. chirps dimension), azimuth antenna and elevation antenna of the intermediate frequency signal matrix, an echo signal matrix with four dimensions of information including distance, speed, azimuth and elevation is obtained. Specifically,

[0019] By measuring the frequency change of the intermediate frequency signal, the distance between the target and the radar is estimated. The calculation formula of distance R is:

[0020]

[0021] Where c is the speed of light, f IF It is the frequency difference between the transmitted signal and the received signal, that is, the intermediate frequency signal frequency.

[0022] The phase difference of different chirps of the intermediate frequency signal reflects the speed information of the target in the radial direction of the radar. The calculation formula of the speed v is:

[0023]

[0024] Where λ is the wavelength of the radar signal, is the initial phase difference between the transmitted signal and the received signal.

[0025] The calculation formula of the azimuth angle of the target object is:

[0026]

[0027] in, It is the phase difference between the signals received by adjacent horizontal receiving antennas. The estimation of the elevation angle φ in the vertical direction is similar.

[0028] At this time, the radar signal matrix contains target echo signals, environmental noise, and various interference signals, which need to be further processed to remove interference and extract effective information on the behavior of multiple human targets from the radar signal matrix.

[0029] Step 2: Through two-dimensional CFAR detection, the virtual power target is suppressed to obtain a relatively pure signal matrix, suppress the thermal noise in the signal and detect the target unit of interest. The target unit of interest is transformed from the range, azimuth and elevation angle domains to the spatial coordinate domain.

[0030] x=R·cosφ·sinθ (4a)

[0031] y=R·cosφ·cosθ (4b)

[0032] z=R·sinφ (4c)

[0033] Among them, x, y, z are the three-dimensional coordinate information of the target, R, θ, φ are the distance, azimuth, and pitch angle of the target. After fusing the original Doppler velocity information v of the target unit, each unit of interest is represented as a four-dimensional vector t i The form of a single point cloud vector.

[0034] t i =[x i y i z i v i ] (5)

[0035] Among them, t i Represents a single target unit feature vector, i is the index of the target unit. Each target unit of interest contained in each frame is extracted and accumulated sequentially in each time frame to form a feature matrix frame based on the frame sequence m .

[0036]

[0037] In this way, the original frame information is converted from a 4-dimensional feature matrix to a 2-dimensional point cloud matrix, which reduces the spatial dimension and shields useless background information, effectively alleviating the sparsity problem of the original feature matrix.

[0038] Step 3: First, use the density clustering algorithm to cluster the point cloud matrix, cluster the same type of point clouds in each frame into a cluster, and mark the point clouds that cannot form a cluster as noise and remove them. Specifically, for each t in a frame i , set the neighborhood distance threshold ε and the sample number threshold MinPts, the ε neighborhood includes the samples in the current frame set and t i The subsample set whose Euclidean distance is not greater than ε is D ε (t i ),Right now

[0039] D ε (t i )={t j ∈frame m |distance(t i ,t j )≤ε} (7)

[0040] Where distance(t i ,t j ) is the point cloud vector t i With t j The Euclidean distance between them is used to divide this subsample set D ε (ti ) is denoted as |D ε (t i )|, if|D ε (t i )|>MinPts, then t i Is the core object, D ε (t i ) are marked as a cluster. Then the point clouds belonging to the same cluster are averaged to convert a cluster of point clouds into a single point cloud point.

[0041] Secondly, the local outlier factor (LOF) algorithm is used to detect outliers on the target point cloud and filter out outliers that exceed the application range of human targets. The LOF algorithm mainly includes three steps:

[0042] Step 1: Calculate the kth reachable distance of each point in the kth distance neighborhood of each point cloud.

[0043] reach_distance k (t i ,t j )=max{distance k (t i ),distance(t i ,t j )} (8)

[0044] where distance k (t i ) is the domain point cloud t i The kth distance, distance(t i ,t j ) is the neighborhood point t i To point t j distance.

[0045] Step 2: Calculate the local kth local reachability density lrd of each point cloud through the kth reachable distance of each point cloud k

[0046]

[0047] Where N k (t j ) is the point cloud t j The k-th distance neighborhood of .

[0048] Step 3: After obtaining the local reachable density of each point, calculate the kth local outlier factor LOF of each point cloud

[0049]

[0050] According to the set LOF threshold, point clouds with LOF values ​​greater than the threshold are marked as outliers and filtered out, and normal point clouds with LOF values ​​less than or equal to the threshold are retained.

[0051] Finally, based on the position information of multiple human targets, the Hungarian algorithm is used to match all point clouds containing multiple human target behaviors in sequence according to the time frame order. By calculating the "cost" between the point cloud and the initial target information, the point cloud with the smallest "cost" weight is divided into the same target. Relying on the continuity of human behavior, the point cloud is tracked based on human targets, and finally the multi-person point cloud matrix is ​​divided into a single-person point cloud matrix independent of other targets.

[0052] Step 4: Build an action recognition network based on a multi-dimensional point cloud neural network, input the pure single-target point cloud matrix obtained in step 3 into the neural network for training, adjust the network parameters, and perform recognition after the neural network converges.

[0053] The input of the network is a single-target point cloud matrix with an unfixed number of point clouds, and each point cloud matrix contains multimodal information within the duration of the action. First, the network uses 2D extended convolution to extract the feature information of adjacent point cloud vectors, including the horizontal feature matrix information of the point cloud vector dimension and the vertical temporal feature information, and expands the multimodal feature information horizontally. After three layers of extended convolution for feature extraction, the original input point cloud of size n×p is expanded to a feature matrix of n×256 or higher dimensions, which contains multi-dimensional fusion information of human behavior. Then, the feature matrix of size n×256 is input to the GlobalMaxPool layer for global maximum pooling to generate a feature vector of size 1×256. In this way, the network processes point cloud matrices with different numbers of point clouds into feature vectors of the same size, and the maximum pooling can preferentially extract the most significant features in the feature matrix. Finally, the fused feature vector is sent to a three-layer fully connected neural network for classification, and finally activated by the softmax function to obtain a result vector of size 1×k, where k is the number of types of output point clouds. After each convolution operation, a batch normalization layer is used to normalize the function mapping, improve the efficiency of neural network parameter adjustment, and make the model easier to converge.

Claims

1. A method for multi-human target behavior recognition based on millimeter wave radar, characterized in that The following steps are involved: Step 1: Use frequency modulated continuous wave (FMCW) radar to collect radar signals of multiple human target behaviors. The radar echo signal is mixed with the transmission signal to obtain an intermediate frequency signal containing target information. By performing four Fast Fourier Transforms (FFT) on the fast time (i.e., sampling point dimension), slow time (i.e., chirps dimension), azimuth antenna, and elevation antenna of the intermediate frequency signal matrix, an echo signal matrix with four dimensions of information including distance, speed, azimuth, and elevation is obtained. Step 2: Use the two-dimensional cell averaging constant false-alarm rate (CA-CFAR) algorithm to suppress the virtual power target to obtain a relatively pure signal matrix, suppress the thermal noise in the signal and detect the target cell of interest. Convert the target cell of interest from the range, azimuth and elevation angle domains to the spatial coordinate domain. x=R·cosφ·sinθ (1a) y=R·cosφ·cosθ (1b) z=R·sinφ (1c) Among them, x, y, z are the three-dimensional coordinate information of the target, R, θ, φ are the distance, azimuth, and pitch angle of the target. After fusing the original Doppler velocity information v of the target unit, each unit of interest is represented as a four-dimensional vector t i The form of a single point cloud vector. t i =[x i y i z i v i ] (2) Among them, t i Represents a single target unit feature vector, i is the index of the target unit. Each target unit of interest contained in each frame is extracted and accumulated sequentially in each time frame to form a feature matrix frame based on the frame sequence m . In this way, the original frame information is converted from a 4-dimensional feature matrix to a 2-dimensional point cloud matrix, which reduces the spatial dimension and shields useless background information, effectively alleviating the sparsity problem of the original feature matrix. Step 3: First, use the density-based spatial clustering algorithm (DBSCAN) to cluster the point cloud matrix, cluster the same type of point clouds in each frame into a cluster, and mark the point clouds that cannot form a cluster as noise and remove them. Specifically, for each t in a frame i , set the neighborhood distance threshold ε and the sample number threshold MinPts, the ε neighborhood includes the samples in the current frame set and t i The subsample set whose Euclidean distance is not greater than ε is D ε (t i ),Right now D ε (t i )={t j ∈frame m |distance(t i ,t j )≤ε} (4) Where distance(t i ,t j ) is the point cloud vector t i With t j The Euclidean distance between them is used to divide this subsample set D ε (t i ) is denoted as |D ε (t i )|, if|D ε (t i )|>MinPts, then t i Is the core object, D ε (t i ) are marked as a cluster. Then the point clouds belonging to the same cluster are averaged to convert a cluster of point clouds into a single point cloud point. Secondly, the local outlier factor (LOF) algorithm is used to detect outliers on the target point cloud and filter out outliers that exceed the application range of human targets. The LOF algorithm mainly includes three steps: Step 1: Calculate the kth reachable distance of each point in the kth distance neighborhood of each point cloud. reach_distance k (t i ,t j )=max{distance k (t i ),distance(t i ,t j )} (5) where distance k (t i ) is the domain point cloud t i The kth distance, distance(t i ,t j ) is the neighborhood point t i To point t j distance. Step 2: Calculate the local kth local reachability density lrd of each point cloud through the kth reachable distance of each point cloud k Where N k (t j ) is the point cloud t j The k-th distance neighborhood of . Step 3: After obtaining the local reachable density of each point, calculate the kth local outlier factor LOF of each point cloud According to the set LOF threshold, point clouds with LOF values ​​greater than the threshold are marked as outliers and filtered out, and normal point clouds with LOF values ​​less than or equal to the threshold are retained. Finally, based on the position information of multiple human targets, the Hungarian algorithm is used to match all point clouds containing multiple human target behaviors in sequence according to the time frame order. By calculating the "cost" between the point cloud and the initial information of the target, the point cloud with the smallest "cost" weight is divided into the same target. Relying on the continuity of human behavior, the point cloud is tracked based on human targets, and finally the multi-person point cloud matrix is ​​divided into a single-person point cloud matrix independent of other targets. Step 4: Build an action recognition network based on a multi-dimensional point cloud neural network, input the pure single-target point cloud matrix obtained in step 3 into the neural network for training, adjust the network parameters, and perform recognition after the neural network converges. The input of the network is a single-target point cloud matrix with an unfixed number of point clouds, and each point cloud matrix contains multimodal information within the duration of the action. First, the network uses 2D extended convolution to extract the feature information of adjacent point cloud vectors, including the horizontal feature matrix information of the point cloud vector dimension and the vertical temporal feature information, and expands the multimodal feature information horizontally. After three layers of extended convolution for feature extraction, the original input point cloud of size n×p is expanded to a feature matrix of n×256 or higher dimensions, which contains multi-dimensional fusion information of human behavior. Then, the feature matrix of size n×256 is input to the GlobalMaxPool layer for global maximum pooling to generate a feature vector of size 1×256. In this way, the network processes point cloud matrices with different numbers of point clouds into feature vectors of the same size, and the maximum pooling can preferentially extract the most significant features in the feature matrix. Finally, the fused feature vector is sent to a three-layer fully connected neural network for classification, and finally activated by the softmax function to obtain a result vector of size 1×k, where k is the number of types of output point clouds. After each convolution operation, a batch normalization layer is used to normalize the function mapping, improve the efficiency of neural network parameter adjustment, and make the model easier to converge.

Images