Radar target detection method based on camera supervision feature enhancement

Through information fusion of camera and millimeter-wave radar images and neural network supervision training, combined with 3D convolution and deconvolution layers, the problem of false targets and poor recognition performance in vehicle-mounted radar detection is solved, and the detection accuracy and robustness are improved. All-weather object detection is achieved.

CN119064925APending Publication Date: 2024-12-03HENAN POLYTECHNIC UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411080601.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Vehicle millimeter-wave radar lacks semantic information in target detection, which makes it difficult to distinguish false targets, easily misdetect, and has low resolution, poor recognition performance, and cannot identify target feature information, making it difficult to accurately detect in harsh environments.

Method used

By fusion of information on camera images and millimeter-wave radar images and supervised and assisted training of neural networks, the F-RODNet network model is used, combining 3D convolution and deconvolution layers to perform feature enhancement and data fusion, and non-maximum suppression is used. The algorithm removes redundant target points and improves the accuracy and reliability of radar target detection.

Benefits of technology

Improve the accuracy and system robustness of radar target detection, overcome the problem of redundant false target points in radar images, enhance the detection capability in harsh environments, reduce the dependence on the camera, and realize all-weather detection .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119064925A_ABST
    Figure CN119064925A_ABST
Patent Text Reader

Abstract

The invention discloses a radar target detection method based on camera supervision feature enhancement, and the method comprises the steps: collecting a frequency modulation continuous wave radar echo signal to generate a radar radio frequency image, and detecting a radar peak value; obtaining semantic information in the camera image through the category and position information of the target; a camera image containing target position category information is mapped to a radar peak value containing the target position information according to the position for data fusion, an input confidence map with target feature annotations is obtained, the input confidence map is sent to a network test part for auxiliary training, and an F-RODNet network model is obtained; training a neural network by using the fused target probability graph, wherein an F-RODNet structure is composed of an input layer, a convolutional auto-encoder and a fusion layer; according to a target confidence map output by the neural network, the position and the category of the target are judged by adopting a position-based non-maximum suppression method, and redundant target points are removed. The millimeter wave radar target detection precision and reliability are improved, and the safety performance of the auxiliary driving system is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of radar signal detection, and in particular relates to a radar target detection method based on camera supervision feature enhancement. Background Art

[0002] Intelligent assisted driving systems require a detection method that can accurately and robustly detect targets during driving and is not affected by weather and lighting. Most detection schemes use multi-sensor fusion to improve the accuracy and reliability of perception results. Commonly used sensors include cameras, lidars, and millimeter-wave radars. The cost of millimeter-wave radars is much lower than that of lidars, and they are less affected by lighting and weather. They have a longer range, high ranging accuracy, and good stability, and have great utilization value. However, the current millimeter-wave radar has a relatively low resolution and poor recognition performance. It cannot identify the characteristic information of the target, and it is difficult to annotate semantic information. In multi-sensor fusion, the point cloud of the millimeter-wave radar is very sparse and contains limited information, so it is mainly used to supplement the camera. This type of method only uses the positioning information in the radar signal, does not solve the detection problem under visual constraints, and is prone to missed detection and false detection. Summary of the invention

[0003] The purpose of the present invention is to address the problem that vehicle-mounted millimeter-wave radar lacks semantic information such as target category and position, which makes it difficult to distinguish false targets and is prone to false detection. A radar target detection method based on camera supervised feature enhancement is provided. By fusing camera images and millimeter-wave radar images and performing supervised auxiliary training on neural networks, the millimeter-wave radar target detection accuracy and reliability are improved, thereby improving the safety performance of the assisted driving system.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A radar target detection method based on camera supervised feature enhancement includes the following steps:

[0006] Step 1: Use a millimeter wave radar sensor to collect frequency modulated continuous wave radar echo signals, use distance angle fast Fourier transform to determine the target azimuth information, generate a radar radio frequency image, and use a constant false alarm probability detector to detect the radar peak;

[0007] Step 2, using a monocular camera sensor to collect images to obtain semantic information in the camera image, including the category and location information of the target;

[0008] Step 3: Map the camera image containing the target location category information to the radar peak containing the target location information according to the position, enter the data fusion stage for fusion operation, obtain the input confidence map with target feature annotations, and send it to the network test part for auxiliary training to obtain the F-RODNet (Feature-enhanced Radar Object Detection) network model;

[0009] Step 4: Use the fused target probability map to train the neural network. The F-RODNet structure consists of an input layer, a convolutional autoencoder, and a fusion layer. The input layer is the radar RF image input to be tested. 3D convolution is used to initially extract features, and 3D deconvolution is used to restore the data size before convolution.

[0010] Step 5: According to the target confidence map output by the neural network, the position and category of the target are determined by using the position-based non-maximum suppression method, and redundant target points are removed.

[0011] Furthermore, the specific steps of step 1 include:

[0012] Step 1-1, the synthesizer generates a linear frequency modulation signal, which is sent out by the transmitting antenna and received by the radar receiving antenna after being reflected by the object;

[0013] Step 1-2: The mixer combines the received signal and the transmitted signal together and generates raw radar data through an analog-to-digital converter;

[0014] In steps 1-3, after performing a fast Fourier transform of the range angle on the radar raw data, a low-pass filter is used to remove the high-frequency noise in all chirps at a rate of 30 FPS, and finally the radar RF image data is generated for subsequent camera radar fusion supervision and training network models.

[0015] Further, the specific steps of step 2 include:

[0016] Step 2-1, set sliding windows of different sizes as candidate areas, extract visual features related to the candidate areas, use the Mask R-CNN network as the target category detector for identification and classification, and let the sliding window traverse all areas until the category information of all targets is obtained;

[0017] Step 2-2, use the monocular camera-based visual object 3D positioning system to obtain the position information in the image semantics; extract the absolute scale information through the stereo target self-supervision method in the training image; use adaptive plane estimation to locate unreliable targets in the depth estimation to ensure the accuracy of the depth estimation of each frame; and then use multi-target tracking technology to avoid the problem of missed detection of the target to be measured caused by inter-frame bias.

[0018] Further, the specific steps of step 3 include:

[0019] Step 3-1, project the detected object category and 3D position information from the camera from the camera coordinates to the radar range-azimuth map coordinates through coordinate transformation. The transformation can be expressed as:

[0020]

[0021] Where (ρ c ,θ c ) is the projection position of the radar range-azimuth coordinate system, and c is the target sequence number in each frame; (x c ,z c ) is the position of the target in the camera coordinates; (x o ,z o ) is the position of the radar origin corresponding to the camera coordinates;

[0022] Step 3-2, calculate the probability map of the target position in the camera and radar images. The camera position probability map of the target class is defined by the following formula:

[0023]

[0024] Where i is the ordinal number of the data input frame that generates the probability map, δ (cls) is the typical azimuth error of camera positioning, s (cls) is the target confidence scale constant, i cls is the target category of the i-th frame, d i is the target depth of the i-th frame, x RGB is the input camera image, cls is the target category, c i is the depth confidence, N(·) represents the normalization operation on the probability map of each object; is the target position in the camera coordinate system of the i-th frame; T is the transposition operation;

[0025] Step 3-3, the probability map of the radar target location is defined by the following formula:

[0026]

[0027] Where j is the ordinal number of the data input frame that generates the probability map, is the j-th frame radar range resolution, x R is the input radar image, ε(·) is the radar azimuth resolution; is the projection position of the radar j-th frame range-azimuth coordinate system;

[0028] Step 3-4, use element dot product to obtain the fusion probability of each target class, and use peak detection to obtain the fusion probability P CRF Get the fusion annotation:

[0029]

[0030] In the formula, is the radar target position probability map, P r (x) is the camera target position probability map.

[0031] Further, in step 4, the input layer needs to extract features from the input data, and select a convolution kernel with a relatively small size according to the number of data sets; the input layer is composed of three three-dimensional convolutions and three deconvolutions, and each convolution and deconvolution is followed by batch normalization and ReLU activation functions; the three-dimensional convolution initially extracts the target features, and the deconvolution is used to restore the data size after convolution, so as to prepare the input for the subsequent convolution autoencoder;

[0032] The convolutional autoencoder consists of three-dimensional convolution and batch normalization layers, including six 3D convolutional layers, one SE attention module and three 3D deconvolutional layers; the convolutional autoencoder also includes a dropout layer, which randomly discards some data to prevent overfitting;

[0033] The fusion layer includes a fusion module, which adopts multi-scale fusion, connects all features along the channel dimension, and inputs them into the deep network for classification decision. There are two layers after the feature fusion module, one is a three-dimensional initial layer, including three convolution kernels (3, 5, 5), (3, 3, 3), and (3, 1, 21). The first two convolution kernels allow the network to utilize multi-level feature extraction, generally extracting features of size 5×5 and local features of size 3×3. The third convolution kernel is used to promote the neural network to observe a larger area in terms of angle; the other is an ordinary three-dimensional convolution layer with a convolution kernel size of (3, 3, 3).

[0034] Further, the step 5 is specifically as follows:

[0035] Step 5-1, perform peak detection on the neural network output confidence map to detect all peaks within the 3×5 window;

[0036] Step 5-2, determine the number of peaks. If one peak is detected, there is only one target. If more than one peak is detected, the peak type is determined, including pedestrians, bicycles and cars.

[0037] Step 5-3, remove redundant target points according to the label information of adjacent peaks. If the peak labels of adjacent targets are different, directly output the confidence map containing target information; if the peak label types of adjacent targets are the same, calculate the position similarity OLS between peaks, the formula is:

[0038]

[0039] Where d is the distance between two target points in the RF image; s is the distance from the target to the radar sensor, indicating the target distance position information; k cls The size of each target class is determined by the average size of the corresponding class target, so that OLS is reasonably distributed between 0 and 1; OLS is regarded as a Gaussian distribution, where the distance d is used as the deviation and s 2 k cls as variance;

[0040] The threshold is empirically determined to be 0.3, so that the category of the target can be accurately determined. If the OLS is greater than the threshold, it is judged that the similarity is too high and is merged into the same target. If the OLS is lower than the threshold, it is divided into different targets and a confidence map containing target information is output.

[0041] The beneficial effects of the present invention are:

[0042] 1. The present invention makes full use of information fusion technology, combines the target detection and three-dimensional positioning method of the monocular camera with the synchronously calibrated millimeter-wave radar frame, so that the millimeter-wave radar radio frequency image contains the target category and position information, thereby improving the accuracy of target detection and the robustness of the detection system;

[0043] 2. The present invention adds a time-initial convolution layer and uses continuous radar frames instead of a single frame as system input, making better use of timing information, achieving more accurate detection, and overcoming the problems of blurring and shielding that may occur in single-frame detection;

[0044] 3. The present invention uses a three-dimensional convolutional neural network to process the frequency and phase information of radio signals, solving the problem that the typical real-number CNN architecture does not match the radio signals;

[0045] 4. The target position similarity is used to implement the position-based non-maximum suppression algorithm. By calculating the target position similarity of the distance category and scale information correlation of the two detected targets on the confidence map, the problem of redundant false target points being easily generated due to the lack of bounding box definition in the radar image is overcome, thereby improving the accuracy of radar target detection.

[0046] 5. The present invention can accurately identify targets in harsh environments, improve the detection accuracy in strong and weak lighting environments, reduce the dependence of the autonomous driving system on cameras, and at the same time has the characteristics of all-weather detection and detection of dynamic targets, and the performance is more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the method of the present invention;

[0048] Figure 2 A flow chart of generating a radio frequency image of the present invention;

[0049] Figure 3 It is a schematic diagram of the network structure of the present invention;

[0050] Figure 4 Schematic diagram of the position-based non-maximum suppression algorithm of the present invention;

[0051] Figure 5a This is a schematic diagram of the experimental environment with good light and sight;

[0052] Figure 5b This is a schematic diagram of the experimental environment with poor light and visibility;

[0053] Figure 5c This is a schematic diagram of the vehicle low-speed driving experimental environment;

[0054] Figure 5d Schematic diagram of the vehicle high-speed driving experimental environment. DETAILED DESCRIPTION

[0055] The present invention is described in detail below with reference to the accompanying drawings and specific implementation examples.

[0056] like Figure 1 As shown, the present invention is a radar target detection method based on camera supervised feature enhancement, and the system framework is divided into two parts: data preparation and network testing. In the data preparation part, firstly, the camera and millimeter wave radar are used to collect images of the training target respectively, and the semantic information such as the position and category information of the target in the camera image is detected by the 3D target position detection method, and the peak value of the radar image is detected by the constant false alarm detector to obtain the position information of the quasi-target in the radar image, and the information of the two sensors is fused to obtain the input confidence map (pre-ConfMaps) with target feature annotations, which is sent to the network testing part for auxiliary training to obtain the F-RODNet network model. The F-RODNet proposed by the present invention integrates the target information of the camera image in the training stage, and only uses the radar radio frequency data to test the model in the testing stage, which can effectively solve the problem of insufficient target recognition accuracy in harsh environments and limited light conditions, and can make up for the defects of low resolution and low recognition accuracy of millimeter wave radar, and improve the detection efficiency of target categories and positions in various environments. In the network test part, the F-RODNet network model predicts the confidence map (out-ConfMaps) of the object target in the RF image to be tested, and uses the position-based non-maximum suppression method to infer the object target category and position.

[0057] In the data preparation part, radar data uses the most common RF images. Figure 2As shown in the figure, the steps of generating raw data are as follows: first, the synthesizer generates a linear frequency modulation signal, which is emitted by the transmitting antenna and received by the receiving antenna after being reflected by the object; the mixer combines the receiving signal and the transmitting signal together and generates raw radar data through the analog-to-digital converter; after performing a fast Fourier transform of the distance angle on the raw radar data, a low-pass filter is used to remove the high-frequency noise in all chirps at a rate of 30FPS, and finally the radar RF image data is generated for subsequent camera radar fusion supervision and training network models.

[0058] A single radar signal lacks semantic information such as target category and location, which can easily lead to false positive detections. Camera feature fusion can make up for this deficiency.

[0059] The present invention uses an effective and robust visual object 3D positioning system based on a monocular camera to obtain the target position information of the camera. First, sliding windows of different sizes are set as candidate areas, and visual features related to the candidate areas are extracted. The MaskR-CNN network is used as a target category detector for identification and classification, and the sliding window is allowed to traverse all areas until the category information of all targets is obtained; then the visual object 3D positioning system based on a monocular camera is used to obtain the position information in the image semantics; the absolute scale information is extracted through the stereoscopic target self-supervision method in the training image; the unreliable targets in the depth estimation are located using adaptive plane estimation to ensure the accuracy of the depth estimation of each frame; and the multi-target tracking technology is used to avoid the problem of missed detection of the target to be measured caused by inter-frame bias.

[0060] The label information obtained by fusing the radar RF image generated in the above steps with the camera image is provided to the neural network. The radar has a strong distance estimation capability. The constant false alarm detection technology is used to detect the peak information in the RF image, and the target peaks that may exist in the radar are fused with the target position categories contained in the camera image. The detection object category and 3D position information from the camera are projected from the camera coordinates to the radar range-azimuth map coordinates through coordinate transformation. The transformation can be expressed as:

[0061]

[0062] Where (ρ c ,θ c ) is the projection position of the radar range-azimuth coordinate system, and c is the target sequence number in each frame; (x c ,z c ) is the position of the target in the camera coordinates; (x o ,z o ) is the position of the radar origin in the camera coordinate system.

[0063] After the camera and radar coordinates are aligned, two probability maps are generated for the camera and radar positions respectively, and then the probability maps are fused by element-wise product to achieve annotation estimation of the input RF image.

[0064] The camera position probability map of the target class is defined as follows:

[0065]

[0066] Where i is the ordinal number of the data input frame that generates the probability map, δ (cls) is the typical azimuth error of camera positioning, s (cls) is the target confidence scale constant, i cls is the target category of the i-th frame, d i is the target depth of the i-th frame, x RGB is the input camera image, cls is the target category, c i is the depth confidence, N(·) represents the normalization operation on the probability map of each object, is the target position in the camera coordinate system of the i-th frame, and T is the transposition operation.

[0067] The probability map of the radar location is defined by:

[0068]

[0069] Where j is the ordinal number of the data input frame that generates the probability map, is the j-th frame radar range resolution, x R is the input radar image, ε(·) is the radar azimuth resolution; is the projection position of the target in the range-azimuth coordinate system of the radar frame j.

[0070] The fusion probability of each target class is obtained by element dot product, and the fusion probability P is obtained by peak detection. CRF Get the fusion annotation:

[0071]

[0072] In the formula, is the radar target position probability map, P r (x) is the camera target position probability map.

[0073] The fused target probability map is used to train the neural network, such as Figure 3 As shown in Figure 1, the F-RODNet structure consists of an input layer, a convolutional autoencoder, and a fusion layer. The front-end data input part is the radar RF image input to be tested. 3D convolution is used to initially extract features, and 3D deconvolution is used to restore the data size before convolution.

[0074] The input layer needs to extract features from the input data. According to the number of data sets, a relatively small convolution kernel is selected to reduce the complexity while providing sufficient spatial movement and trade-off freedom. The input layer consists of three three-dimensional convolutions and three inverse three-dimensional convolutions, and each convolution and deconvolution is followed by batch normalization and ReLU activation functions to speed up the training and convergence of the network, prevent gradient explosion and overfitting, and improve the generalization ability of the neural network. The three three-dimensional convolutions initially extract the target features, and the deconvolution restores the data size after convolution to prepare for the input of the subsequent convolutional autoencoder.

[0075] Convolution and normalization operations are performed in time and space, and the time information of the input signal can be retained after each convolution operation, so as to more effectively retain the characteristics of the target, form a feature layer, and send it to the convolutional autoencoder. The three convolutional autoencoders are the main body of the F-RODNet network, and a dropout layer is added to randomly discard some data to prevent overfitting. The three convolutional encoders focus on extracting the camera, radar, and confidence map label features in the fused data, and are fused after the deconvolution decoder, which can effectively enhance the neural network feature extraction and reconstruction capabilities.

[0076] The convolutional autoencoder consists of three-dimensional convolution and batch normalization layers, including 6 3D convolutional layers, an SE attention module, and 3 3D deconvolutional layers. For the convolutional layers of the convolutional autoencoder, larger convolution kernels are selected to accurately capture the local structure and information in the features. The dropout layer is added to the convolutional autoencoder to randomly discard some data to prevent overfitting. The model can effectively summarize the spatiotemporal patterns of the original data into high-level semantics. The encoder formed by 6 3D convolutional layers compresses high-dimensional inputs into low-dimensional latent codes and extracts the most relevant information from them, while the decoder formed by 3 3D deconvolutional layers decompresses the encoded data and recreates the original input. The purpose of this architecture is to maximize information while minimizing reconstruction errors during encoding.

[0077] Strictly speaking, the convolutional autoencoder is a special case of the traditional autoencoder. The convolutional autoencoder uses convolutional layers and pooling layers instead of traditional fully connected layers. Assuming there are k convolution kernels ω, each with a bias b, and convolution with the input x to generate k feature maps h, the activation function is σ, then the convolution formula is as follows:

[0078] h k =σ(x*ω k +b k )

[0079] Each feature map h is convolved with the transpose of its corresponding convolution kernel and the results are summed, plus the bias c. The activation function is still σ, so the deconvolution formula is as follows:

[0080]

[0081] By comparing the Euclidean distance between the input sample and the final result reconstructed using the features and optimizing it through the back-propagation algorithm, a complete convolutional autoencoder can be obtained.

[0082] The fusion layer includes a fusion module. The fusion module uses multi-scale fusion to connect all features along the channel dimension and input them into the deep network for classification decision. There are two layers after the feature fusion module. One is a three-dimensional initial layer, including three convolution kernels (3, 5, 5), (3, 3, 3), (3, 1, 21). The first two convolution kernels allow the network to use multi-level feature extraction, generally extracting 5×5 features and local 3×3 features. The third convolution kernel is used to push the neural network to observe a larger area in terms of angle. The other is a common three-dimensional convolution layer with a convolution kernel size of (3, 3, 3). The three-dimensional initial layer performs channel, range and angle dimension operations on the fused features, and the common convolution layer performs time, range and angle dimension operations on the fused features. To avoid the collapse of the time dimension on the three-dimensional initial layer, the operation is repeated for each timestamp, and all initial results are connected along the time dimension. This fusion method can capture the details and features of the target object at different scales. By fusing information at different scales, the performance of target detection and recognition can be improved, making the model more robust and accurate. The convolutional autoencoder provides different upper and lower layer information. The effective integration of multi-scale fusion makes this information more comprehensive and richer, which is crucial for understanding the relationship between objects and scenes in images. This fusion method can improve the robustness and generalization ability of the model, and has better adaptability to lighting changes in different scenes, thereby improving the application effect and stability of the model.

[0083] The essence of the attention mechanism is that the network automatically adjusts the weights of different elements according to the input data to achieve a weighted distribution process for the input data. The attention mechanism can be combined with the convolutional neural network to refine features and filter information. In the channel dimension, the attention mechanism can be used to obtain the importance of each channel of the feature map, and then assign another weight value to each feature, so that the neural network can focus on certain feature channels, enhance feature channels that are useful for the current task, and suppress feature channels that are not very useful for the current task. Figure 3The SE attention input feature module in is divided into two paths, one to generate channel attention weights, and the other to update and refine the original features. In the branch that generates channel attention weights, the features are compressed along the spatial dimension by average pooling, and then learned through a double convolution layer. The batch normalization layer accelerates the convergence speed of the network, improves the generalization ability of the network and suppresses overfitting. After connecting the ReLU and Sigmoid activation functions, the channel attention weight matrix is ​​generated, and the matrix is ​​multiplied with the original features in the other path to further refine the features.

[0084] After the confidence map is output from the network, it is necessary to use a position-based non-maximum suppression algorithm for post-processing to obtain the final detection result. The traditional NMS algorithm is often used in image-based target detection to remove redundant bounding boxes from the detection results. The traditional NMS uses the intersection ratio union as a criterion to determine whether the bounding box should be deleted because it overlaps too much with the detection candidate with the highest confidence. However, the targets in the RF image have no bounding box definition, and no result output ConfMaps. The present invention adopts a similarity metric called Object Location Similarity (OLS), which is similar to the role of the intersection ratio union. OLS is used to remove redundant target points and describe the correlation of the distance, category and scale information of two detected objects on ConfMaps. The OLS formula is as follows:

[0085]

[0086] Where d is the distance between two target points in the RF image; S is the distance from the target to the radar sensor, indicating the target distance position information; k cls The size of each target class can be determined by the average size of the corresponding class target, so that OLS is reasonably distributed between 0 and 1. OLS is regarded as a Gaussian distribution, where the distance d is used as the deviation and d is used as the variance. The threshold is empirically determined to be 0.3, so that the category of the target can be accurately determined. If OLS is greater than the threshold, it is judged that the similarity is too high and merged into the same target. If OLS is lower than the threshold, it is divided into different targets and a confidence map containing target information is output.

[0087] OLS is a similarity measure that takes into account both the size and distance of objects, so it is more reasonable than other traditional distance measures such as Euclidean distance and Mahalanobis distance. Figure 4 shown.

[0088] The process of removing redundant target points can be summarized as follows: perform peak detection on the neural network output confidence map to detect all peaks in the 3×5 window; determine the number of peaks, if one peak is detected, there is only one target, if it is greater than or equal to two, perform peak type determination, including three types of pedestrians, bicycles and cars; remove redundant target points based on the label information of adjacent peaks. If the peak labels of adjacent targets are different, directly output the confidence map containing target information; if the peak label types of adjacent targets are the same, calculate the position similarity OLS between the peaks. If the OLS is greater than the threshold, it is judged that the similarity is too high and merged into the same target. If the OLS is lower than the threshold, it is divided into two different targets and the confidence map containing target information is output.

[0089] In addition, during the test phase, continuous radar RF segments can be sent to the network so that the erroneous information in a certain frame can be corrected by the radar segment of the previous frame to improve the performance of the system. At the same time, inputting continuous frames can increase the speed of network training and testing, which provides different ConfMaps predictions for a single radar frame. Then, these different ConfMaps are averaged and label information is added to the target point to obtain the final ConfMaps result. This solution can improve the robustness of the system and improve both the performance and speed of the system.

[0090] like Figure 5a and Figure 5b As shown in the figure, the system recognition effect diagrams under different light intensity scenes are listed respectively. Figure 5a The experiment was conducted in an open parking lot and in an environment with sufficient light. For such scenes, the target detection system dominated by cameras can easily obtain good detection results, and the detection effect dominated by radar in the present invention is also close to that of cameras. Figure 5b The image is blocked by buildings and trees, and the lighting conditions are poor, so the camera target detection is slightly insufficient. The solution of the present invention achieves good detection performance. Figure 5c and Figure 5d As shown in the figure, the detection effects at different target speeds are listed respectively. Figure 5c The system achieves good detection results in low-speed environments for vehicles equipped with radar. Figure 5d For the detection effect of a vehicle equipped with radar when traveling quickly, the system detection effect is reduced in high-speed scenarios. The target detection system of the present invention can also maintain a high accuracy rate in fast scenarios, providing a safer guarantee for the field of intelligent driving.

Claims

1. A radar target detection method based on camera supervised feature enhancement, characterized in that: The steps include: Step 1: Use a millimeter wave radar sensor to collect frequency modulated continuous wave radar echo signals, use distance angle fast Fourier transform to determine the target azimuth information, generate a radar radio frequency image, and use a constant false alarm probability detector to detect the radar peak; Step 2, using a monocular camera sensor to collect images to obtain semantic information in the camera image, including the category and location information of the target; Step 3: Map the camera image containing the target location category information to the radar peak containing the target location information according to the position, enter the data fusion stage for fusion operation, obtain the input confidence map with target feature annotations, and send it to the network test part for auxiliary training to obtain the F-RODNet network model; Step 4: Use the fused target probability map to train the neural network. The F-RODNet structure consists of an input layer, a convolutional autoencoder, and a fusion layer. The input layer is the radar RF image input to be tested. 3D convolution is used to initially extract features, and 3D deconvolution is used to restore the data size before convolution. Step 5: According to the target confidence map output by the neural network, the position and category of the target are determined by using the position-based non-maximum suppression method, and redundant target points are removed.

2. The radar target detection method based on camera supervised feature enhancement according to claim 1, characterized in that: The specific steps of step 1 include: Step 1-1, the synthesizer generates a linear frequency modulation signal, which is sent out by the transmitting antenna and received by the radar receiving antenna after being reflected by the object; Step 1-2: The mixer combines the received signal and the transmitted signal together and generates raw radar data through an analog-to-digital converter; In steps 1-3, after performing a fast Fourier transform of the range angle on the radar raw data, a low-pass filter is used to remove the high-frequency noise in all chirps at a rate of 30 FPS, and finally the radar RF image data is generated for subsequent camera radar fusion supervision and training network models.

3. The radar target detection method based on camera supervised feature enhancement according to claim 1, characterized in that: The specific steps of step 2 include: Step 2-1, set sliding windows of different sizes as candidate areas, extract visual features related to the candidate areas, use the Mask R-CNN network as the target category detector for identification and classification, and let the sliding window traverse all areas until the category information of all targets is obtained; Step 2-2, use the monocular camera-based visual object 3D positioning system to obtain the position information in the image semantics; extract the absolute scale information through the stereo target self-supervision method in the training image; use adaptive plane estimation to locate unreliable targets in the depth estimation to ensure the accuracy of the depth estimation of each frame; and then use multi-target tracking technology to avoid the problem of missed detection of the target to be measured caused by inter-frame bias.

4. The radar target detection method based on camera supervised feature enhancement according to claim 1, characterized in that: The specific steps of step 3 include: Step 3-1, project the detected object category and 3D position information from the camera from the camera coordinates to the radar range-azimuth map coordinates through coordinate transformation. The transformation can be expressed as: Where (ρ c ,θ c ) is the projection position of the radar range-azimuth coordinate system, and c is the target sequence number in each frame; (x c ,z c ) is the position of the target in the camera coordinates; (x o ,z o ) is the position of the radar origin corresponding to the camera coordinates; Step 3-2, calculate the probability map of the target position in the camera and radar images. The camera position probability map of the target class is defined by the following formula: Where i is the ordinal number of the data input frame that generates the probability map, δ (cls) is the typical azimuth error of camera positioning, s (cls) is the target confidence scale constant, i cls is the target category of the i-th frame, d i is the target depth of the i-th frame, x RGB is the input camera image, cls is the target category, c i is the depth confidence, N(·) represents the normalization operation on the probability map of each object; is the target position in the camera coordinate system of the i-th frame; T is the transposition operation; Step 3-3, the probability map of the radar target location is defined by the following formula: Where j is the ordinal number of the data input frame that generates the probability map, is the j-th frame radar range resolution, x R is the input radar image, ε(·) is the radar azimuth resolution; is the projection position of the radar j-th frame range-azimuth coordinate system; Step 3-4, use element dot product to obtain the fusion probability of each target class, and use peak detection to obtain the fusion probability P CRF Get the fusion annotation in: In the formula, is the radar target position probability map, P r (x) is the camera target position probability map.

5. The radar target detection method based on camera supervised feature enhancement according to claim 1, characterized in that: In step 4, the input layer needs to extract features from the input data, and select a convolution kernel with a relatively small size according to the number of data sets; the input layer is composed of three three-dimensional convolutions and three deconvolutions, and each convolution and deconvolution is followed by batch normalization and ReLU activation functions; the three-dimensional convolution initially extracts the target features, and the deconvolution is used to restore the data size after convolution, so as to prepare the input for the subsequent convolution autoencoder; The convolutional autoencoder consists of three-dimensional convolution and batch normalization layers, including six 3D convolutional layers, one SE attention module and three 3D deconvolutional layers; the convolutional autoencoder also includes a dropout layer, which randomly discards some data to prevent overfitting; The fusion layer includes a fusion module, which adopts multi-scale fusion, connects all features along the channel dimension, and inputs them into the deep network for classification decision. There are two layers after the feature fusion module, one is a three-dimensional initial layer, including three convolution kernels (3, 5, 5), (3, 3, 3), and (3, 1, 21). The first two convolution kernels allow the network to utilize multi-level feature extraction, generally extracting features of size 5×5 and local features of size 3×3. The third convolution kernel is used to promote the neural network to observe a larger area in terms of angle; the other is an ordinary three-dimensional convolution layer with a convolution kernel size of (3, 3, 3).

6. The radar target detection method based on camera supervised feature enhancement according to claim 1, characterized in that: The step 5 is specifically as follows: Step 5-1, perform peak detection on the neural network output confidence map to detect all peaks within the 3×5 window; Step 5-2, determine the number of peaks. If one peak is detected, there is only one target. If more than one peak is detected, the peak type is determined, including pedestrians, bicycles and cars. Step 5-3, remove redundant target points according to the label information of adjacent peaks. If the peak labels of adjacent targets are different, directly output the confidence map containing target information; if the peak label types of adjacent targets are the same, calculate the position similarity OLS between peaks, the formula is: Where d is the distance between two target points in the RF image; s is the distance from the target to the radar sensor, indicating the target distance position information; k cls The size of each target class is determined by the average size of the corresponding class target, so that OLS is reasonably distributed between 0 and 1; OLS is regarded as a Gaussian distribution, where the distance d is used as the deviation and s 2 k cls as variance; The threshold is empirically determined to be 0.3, so that the category of the target can be accurately determined. If the OLS is greater than the threshold, it is judged that the similarity is too high and is merged into the same target. If the OLS is lower than the threshold, it is divided into different targets and a confidence map containing target information is output.

Citation Information

Cited By

  • Semantic segmentation detection method and device for marine mammal tick signal and medium

    CN120279554A

  • Target detection method and device based on Leiyu fusion perception and storage medium

    CN121348313A