Human target detection method based on radar signal heat map
By using residual network and feature matrix fusion technology in radar signal heat map, the problem of insufficient accuracy of human object detection in complex environments is solved, and more efficient feature extraction and recognition effects are achieved.
Patent Information
- Application Number
- CN202111273690.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-10-29
AI Technical Summary
The existing radar signal heat map human object detection methods are not very accurate in complex environments, and traditional video detection methods cannot be directly applied to radar signal heat maps, which has problems such as slow recognition speed and insufficient accuracy.
ResNet is used to combine horizontal and vertical heat map inputs to perform data preprocessing and feature matrix reconstruction. Feature fusion is carried out through the feature extraction network and the post-network of ResNet-34 to build an object detection box to realize human object detection.
It improves feature extraction effect and recognition accuracy, simplifies the training process, shortens training time, and achieves higher detection accuracy in complex environments.
Smart Images

Figure CN113989718B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of target image recognition, and in particular relates to a human target detection method oriented to radar signal heat maps. Background Art
[0002] Radar signal heat map, that is, the radio frequency heat map obtained by radar and other radio frequency transceiver devices, is adopted by various detection systems due to its high distance resolution and low difficulty in acquisition, and is widely used in military, transportation, geological engineering and other fields. Extracting and identifying various targets, especially human targets, from radar signal heat maps has become one of the basic requirements and important tasks of various detection systems.
[0003] Human target detection has always been a research hotspot in the field of computer vision. In early studies, target detection methods usually need to extract image features, use the DPM (Deformable Parts Model) model, and use a sliding window to predict the target box (Bounding Box) with a high score. This method is very time-consuming and has low accuracy. Later, researchers introduced deep learning into the field of target detection, which greatly improved the performance of target detection. This type of method can be divided into two categories. One is to divide object recognition and object positioning into two steps, namely Two-stage, such as R-CNN, Fast R-CNN, FasterR-CNN, etc., which has a low recognition error rate but a slow speed, so it is not suitable for real-time detection scenarios. The other is One-stage, which abandons the region proposal method (RegionProposal) and directly generates target coordinate values and classification probabilities. Typical methods include Yolo, SSD, YoloV2, etc. Its recognition speed has been improved, but there are still certain defects in multi-scale prediction and basic classification networks, and the overall accuracy in complex detection scenarios is not ideal. The above studies are mainly aimed at traditional video detection and cannot be directly applied to human target detection in radar signal heat maps. In recent years, universities represented by the Massachusetts Institute of Technology (MIT) have actively explored this field. In 2017, Adib et al. developed an RF-Capture system based on radio frequency signals, which can capture human targets behind walls through walls. In 2018, Zhao et al. developed an RF-Pose system based on RF-Capture that uses WiFi wireless signals for human target detection and key point positioning. These pioneering studies have made good attempts to detect human targets in radar signal heat maps, but there are also many limitations, such as RF-Capture is limited to a single person performing a single action (i.e. walking towards the device), and RF-Pose can only be applied to scenes with a small number of people. Overall, this type of method cannot achieve high accuracy in complex environments. Summary of the invention
[0004] In view of the above problems, the present invention proposes a human target detection method for radar signal heat map, which uses horizontal and vertical heat map inputs at the same time, and introduces residual network ResNet (ResidualNetwork) to improve recognition performance. The method is based on residual network ResNet and uses radar signal heat map as network input. Compared with video images, heat maps can display target information from horizontal and vertical bidirectional perspectives, and effectively remove redundant backgrounds, which is more intuitive. In order to simplify the training process, the method preprocesses the heat map input, which not only integrates the network input parameters and reorganizes the heat map image data, but also helps to improve training efficiency and reduce training time. In the training model, the method extracts the features of the residual network output, and relies on the feature matrix reconstruction operation at the end of the network to fuse the horizontal and vertical heat map features to further improve the feature extraction effect.
[0005] The human target detection method for radar signal heat map includes the following steps:
[0006] Step 1: Data preprocessing; the data includes radio frequency data and video data. The radio frequency data is a radio frequency signal emitted by a multi-antenna array. The signal contacts the target in the target area and reflects back to the origin. The vertical and horizontal bidirectional target space information is obtained according to the reflected signal, so as to construct the original data of the signal heat map, and then perform noise reduction, cropping and labeling. Since the original data is a numerical matrix generated by directly acquiring each frame of radar signal, the 2nd, 4th, 6th and 8th dimensions represent horizontal direction data, and the 1st, 3rd, 5th and 7th dimensions represent vertical direction data. After taking them out respectively, they are processed by image Fourier transform to obtain the heat map matrix, and thus form the horizontal and vertical heat map data sets. The video data is a video obtained by synchronously recording the target scene with a depth camera, and the Faster R-CNN network is used to perform target recognition storage and human body frame extraction on the recorded video image, so as to construct a label set for subsequent network training. 1
[0007] Step 2: Construct a feature extraction network based on ResNet-34; construct a feature extraction network based on the residual block structure in the ResNet network: use the ResNet-34 network structure as the main network, select the Basic Block type for the residual block, and set the number of residual blocks to [3, 4, 6, 3], where the value represents the number of residual blocks contained in the Layer of each residual network; this network is used to extract the feature matrix for subsequent network processing;
[0008] Step 3: Reconstruct the feature matrix; establish a subsequent network architecture after the feature extraction network, reconstruct the feature matrix through matrix flattening, deformation operations, and a fully connected layer to improve the feature matrix;
[0009] Step 4: Fuse the horizontal and vertical heat map features; perform a fusion operation on the feature matrix obtained in the above steps; since the heat map is divided into two data blocks, horizontal and vertical, the block diagram data matrix is obtained after the features of the two heat maps are fused; the additional network architecture constructed by steps 3 and 4 is called the post-network, which processes the output of the feature extraction network;
[0010] Step 5: Construct the target detection frame; use the frame data matrix output by the post-network to perform image segmentation operations on the original image to achieve the visual effect of human target detection.
[0011] Furthermore, the method includes the following steps: in step 1, the video data generates label data through the Fast R-CNN network and performs batch processing operations, thereby extracting the target detection regression box coordinates of each original image, and finally converting the regression box coordinate data into a Tensor data type through the DataLoader method of Pytorch to obtain a label set.
[0012] Furthermore, in step 1, the original data of the signal heat map needs to be subjected to white noise removal to complete the noise reduction process. White noise is generally a value close to 0 or lower than the surrounding data. After completing the above operation, the heat map data is converted into a numerical matrix; in order to complete the cropping and labeling of the original data, the numerical matrix generated by each data frame is cropped and compressed to make it suitable for the network input of step 2; and the matrix representing the horizontal data and the matrix representing the vertical data are marked and distinguished respectively. Finally, the processed data is divided into a training set and a test set in a ratio of 8:2.
[0013] Furthermore, in step 1, for the generated label set, training set and test set, the Pytorch machine learning library is used to encapsulate the data, and finally the encapsulated DataLoader is generated for network loading.
[0014] Furthermore, in step 2, the feature extraction uses four layers as the main architecture, and is supplemented by two different upper and lower pooling layers; for the network input, the first convolution layer and the pooling layer are first used to scale the feature map, and then enter the main unit composed of four layers, each layer is composed of several residual blocks, and every two convolution layers in the main unit are connected by mapping to form a residual block structure; for the network output, it is processed by the global average pooling layer.
[0015] Furthermore, in step 2, the residual block type used in this network is Basic Block, and each residual block consists of two 3x3 convolutional layers.
[0016] Furthermore, in step 3, for the horizontal heat map and the vertical heat map, two feature matrices Out1 and Out2 can be obtained respectively through step 2, and the feature matrices are reconstructed; first, the Flatten operation for multidimensional data is performed, and then the feature processing of the 5x4 fully connected layer FC is performed to achieve matrix optimization; wherein, the Flatten parameters need to be changed to adapt to the different lengths and widths of the network output; after the fully connected layer FC, the feature matrices of the horizontal and vertical heat maps are converted into the final network output; after the fully connected layer, the Reshape operation is used for the test set output display.
[0017] Furthermore, in step 4, the fusion operation includes two different fusion types: Concat and Add; the Add fusion operation is used to complete horizontal and vertical data fusion.
[0018] Furthermore, in step 5, the visual effect is output for each frame of the image; wherein the background of the image is a two-dimensional plane of the original radar signal range, and then the coordinate set data of each regression frame is annotated to the location of the image background to achieve the effect display of human target frame recognition.
[0019] Furthermore, in step 5, the two-dimensional plane of the original radar signal range is defined as an empty plane after the depth information is cut off from the arc area scanned by the radar signal.
[0020] Beneficial effects of the present invention:
[0021] (1) Construct a feature extraction network with a residual network as the main architecture. This method is conducive to solving the degradation problem of deep networks and improving the feature extraction effect. At the same time, the combination of "deep neural network + residual block" is equivalent to a simplified equivalent mapping, which does not generate additional parameters, does not increase the computational complexity, and can still be trained through end-to-end backpropagation.
[0022] (2) Add a post-structure of the feature extraction network. This post-structure is the post-network mentioned above. On the one hand, it is conducive to the optimization and improvement of the feature matrix and improves the accuracy of the final frame recognition. On the other hand, the reconstruction of the matrix makes the fusion of horizontal and vertical heat map features more convenient and faster, shortening the network training time to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 Flow chart of data preprocessing described in an embodiment of the present invention.
[0024] Figure 2 Schematic diagram of the feature extraction network model architecture described in an embodiment of the present invention.
[0025] Figure 3Schematic diagram of the post-network model architecture described in an embodiment of the present invention. DETAILED DESCRIPTION
[0026] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.
[0027] The present invention proposes a target detection method for radar signal heat map. The method is based on the residual network ResNet and uses the radar signal heat map as the network input. Compared with video images, heat maps can display target information from horizontal and vertical bidirectional perspectives, and effectively remove redundant backgrounds, which is more intuitive. In order to simplify the training process, the method preprocesses the heat map input, which not only integrates the network input parameters and reorganizes the heat map image data, but also helps to improve the training efficiency and reduce the training time. In the training model, the method extracts the features of the residual network output, and relies on the feature matrix reconstruction operation at the end of the network to fuse the horizontal and vertical heat map features to further improve the feature extraction effect.
[0028] The target detection method for radar signal heat map of the present invention specifically includes the following steps:
[0029] Step 1 (data preprocessing): The raw data in the present invention includes two categories. One is RF data, which is a RF signal emitted by a multi-antenna array. The signal contacts the target in the target area and reflects back to the origin. The vertical and horizontal bidirectional target space information is obtained based on the reflected signal, and a Fast Fourier Transform (FFT) signal heat map is constructed based on this. The other is video data, which is a video obtained by synchronously recording the target scene using a depth camera. For these two types of data, the present invention will perform Figure 1 The specific steps of the preprocessing shown are as follows.
[0030] (1) Figure 1 As shown in the left branch, the original video data is passed through the Fast R-CNN network to generate label data and perform batch processing operations, so as to extract the target detection regression box coordinates of each original image and finally generate a label set.
[0031] (2) Figure 1 As shown in the right branch, the original heat map is denoised, and then the heat map data is converted into a numerical matrix. Through matrix adjustment operations and channel conversion, a data set suitable for network input is generated, and finally divided into training set and test set.
[0032] (3) For the label set, training set, and test set generated in the above steps, use the Pytorch machine learning library to encapsulate the data and finally generate the encapsulated DataLoader for network loading.
[0033] Step 2 (feature extraction network based on ResNet-34): Figure 2 As shown in the figure, the entire network architecture is optimized based on the basic structure of ResNet34. The network uses four layers as the main architecture, supplemented by two different pooling layers. For the network input, the first convolution layer (7x7) and the pooling layer are used to scale the feature map, and then enter the main unit composed of four layers. Figure 2 The four layers are distinguished according to different grayscale depths, and convolutional layers of the same color depth belong to the same layer. Each layer is composed of several residual blocks. The residual block type used in this network is Basic Block, and each residual block is composed of two 3x3 convolutional layers. Figure 2 In the figure, the two convolutional layers connected by dotted lines constitute a residual block structure. Each dotted line in the figure itself represents the mapping connection of each residual block. This mapping method can back-propagate the gradient to the shallower network layer, which is conducive to building a deeper network architecture. The step size of the first residual block of the second, third, and fourth layers is 2, and the other step sizes are 1. This is to reduce the features for training in the deep network. After the network architecture of four layers, the feature matrix has been reduced to 32 times the original image matrix, thereby completing the feature extraction operation. Finally, through the global average pooling layer (GlobalAverage Pool Layer) processing, the entire network is structurally regularized to prevent overfitting problems. Here, the network architecture of the original ResNet34 ends with a fully connected layer FC. The present invention eliminates the original fully connected layer FC, and instead connects a post-network PostNetwork to the network inputs of the horizontal and vertical heat maps for feature map optimization processing, as described in steps 3 and 4.
[0034] Step 3 (reconstructing feature matrices): For horizontal heatmaps and vertical heatmaps, two feature matrices Out1 and Out2 can be obtained through step 2, which will be Figure 3 The reconstruction process shown in the figure. First, the Flatten operation for multidimensional data is performed, and then the feature processing of the 5x4 fully connected layer FC is performed to achieve matrix optimization. Among them, the Flatten parameters need to be changed to adapt to the different lengths and widths of the network output. After the fully connected layer FC, the feature matrices of the horizontal and vertical heat maps have been converted into the final network output. Since the maximum number of detection targets for target detection is set to 5, and each box coordinate set consists of two coordinates (i.e., 4 values), the parameters of the fully connected layer are 5x4. Finally, in order to keep the data format of the network output and the label set consistent, the Reshape operation is used for the test set output display.
[0035] Step 4 (fusion of horizontal and vertical heat map features): Figure 3 As shown, the horizontal heat map and vertical heat map data, after being processed in step 3, need to undergo feature fusion operation to obtain the final network output. Fusion operations include two different fusion types: Concat and Add. The present invention adopts the Add fusion operation to complete the horizontal and vertical data fusion, that is, to add each element of each dimension of the horizontal and vertical heat maps without changing their size; in contrast, the Concat operation will merge a certain dimension of the above two sets of data and change the output size. Thus, the post-network PostNetwork jointly constituted by step 3 and step 4 is constructed, and the feature extraction network is combined with the network architecture of the post-network to form the entire network system.
[0036] Step 5 (constructing the target detection frame): For the network output of step 4, due to the existence of the fully connected layer FC in step 3, the feature matrix has been converted into the regression frame coordinate data for human detection. Through visualization, each image of the test set can be directly processed to display all the human regression frame diagrams in each image and annotate them in the image. In practical applications, the output of the network can also be directly reflected in the original image, which greatly improves the intuitiveness of the effect.
[0037] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.
Claims
1. A human target detection method based on radar signal heat map, characterized by: The method comprises the following steps: Step 1: Data preprocessing; the data includes radio frequency data and video data. The radio frequency data is a radio frequency signal emitted by a multi-antenna array. The signal contacts the target in the target area and reflects back to the origin. The vertical and horizontal bidirectional target space information is obtained according to the reflected signal, so as to construct the original data of the signal heat map, and then perform noise reduction, cropping and labeling; since the original data is a numerical matrix generated by directly acquiring each frame of radar signal, the 2nd, 4th, 6th and 8th dimensions represent horizontal direction data, and the 1st, 3rd, 5th and 7th dimensions represent vertical direction data, after taking them out respectively, they are processed by image Fourier transform to obtain the heat map matrix, and thus form the horizontal and vertical heat map data sets; the video data is a video obtained by synchronously recording the target scene with a depth camera, and the Faster R-CNN network is used to perform target recognition storage and human body frame extraction on the recorded video image, so as to construct a label set for subsequent network training; Step 2: Build a feature extraction network based on ResNet-34; According to the residual block structure in the ResNet network, the feature extraction network is constructed: the ResNet-34 network structure is used as the main network, the residual block type is selected as the Basic Block type, and the number of residual blocks is set to [3, 4, 6, 3], where the value represents the number of residual blocks contained in the Layer of each residual network; this network is used to extract the feature matrix for subsequent network processing; Step 3: Reconstruct the feature matrix; After the feature extraction network, a subsequent network architecture is established to reconstruct the feature matrix through matrix flattening, deformation operations, and a fully connected layer to improve the feature matrix; In step 3, for the horizontal heat map and the vertical heat map, two feature matrices Out1 and Out2 can be obtained through step 2, and the feature matrices are reconstructed; first, the Flatten operation for multidimensional data is performed, and then the feature processing of the 5x4 fully connected layer FC is performed to achieve matrix optimization; wherein, the Flatten parameters need to be changed to adapt to the different lengths and widths of the network output; after the fully connected layer FC, the feature matrices of the horizontal and vertical heat maps are converted to the final network output; after the fully connected layer, the Reshape operation is used for the output display of the test set; Step 4: Fusion of horizontal and vertical heat map features; The feature matrix obtained in the above steps is fused; since the heat map is divided into two data blocks, horizontal and vertical, the block diagram data matrix is obtained after the features of the two heat maps are fused; the additional network architecture constructed by step 3 and step 4 is called the post-network, which processes the output of the feature extraction network; Step 5: Construct the target detection frame; use the frame data matrix output by the post-network to perform image segmentation operations on the original image to achieve the visual effect of human target detection.
2. The human target detection method based on radar signal heat map according to claim 1, characterized in that: The method comprises the following steps: in step 1, the video data generates label data through the Fast R-CNN network and performs batch processing operations, thereby extracting the target detection regression box coordinates of each original image, and finally converting the regression box coordinate data into a Tensor data type through the DataLoader method of Pytorch to obtain a label set.
3. The human target detection method based on radar signal heat map according to claim 1 is characterized in that: In step 1, the original data of the signal heat map needs to be subjected to white noise removal to complete the noise reduction process. White noise is generally a value close to 0 or lower than the surrounding value. After completing the above operation, the heat map data is converted into a numerical matrix; in order to complete the cropping and labeling of the original data, the numerical matrix generated by each data frame is cropped and compressed to make it suitable for the network input in step 2; The matrices representing horizontal data and vertical data are marked and distinguished respectively; finally, the processed data are divided into training set and test set in a ratio of 8:
2.
4. The human target detection method based on radar signal heat map according to claim 1 is characterized in that: In step 1, for the generated label set, training set, and test set, use the Pytorch machine learning library to encapsulate the data, and finally generate the encapsulated DataLoader for network loading.
5. The human target detection method based on radar signal heat map according to claim 1 is characterized in that: In step 2, the feature extraction uses four layers as the main architecture, supplemented by two different pooling layers; for the network input, the first convolution layer and pooling layer are first used to scale the feature map, and then enter the main unit composed of four layers, each layer is composed of several residual blocks, and every two convolution layers in the main unit are connected by mapping to form a residual block structure; for the network output, it is processed by the global average pooling layer.
6. The human target detection method based on radar signal heat map according to claim 5, characterized in that: In step 2, the residual block type used in this network is Basic Block, and each residual block consists of two 3x3 convolutional layers.
7. The human target detection method based on radar signal heat map according to claim 1 is characterized in that: In step 4, the fusion operation includes two different fusion types: Concat and Add; the Add fusion operation is used to complete horizontal and vertical data fusion.
8. The method for human target detection based on radar signal heat map according to claim 1, characterized in that: In step 5, the visual effect is output for each frame of the image; the background of the image is a two-dimensional plane of the original radar signal range, and then the coordinate set data of each regression frame is marked to the location of the image background to achieve the effect display of human target frame recognition.
9. The human target detection method based on radar signal heat map according to claim 1 is characterized in that: In step 5, the two-dimensional plane of the original radar signal range is defined as an empty plane after the arc area scanned by the radar signal is cut off from the depth information.
Citation Information
Patent Citations
Residual convolutional neural network and PCA dimensionality reduction fused SAR automatic target recognition method
CN108921029A
Radar target recognition method based on two-dimensional weighted residual convolutional neural network
CN112329534A
