A radar three-dimensional target detection method and device based on multi-sensor fusion

Abstract

The present invention belongs to the field of target detection, and discloses a radar three-dimensional target detection method and device based on multi-sensor fusion, the method mainly comprising: acquiring a radar signal, obtaining a radio frequency image of the radar signal, inputting the radio frequency image into an image processing network, obtaining a pre-confidence map of the radio frequency image, calculating the correlation between the detection targets in the pre-confidence map, and obtaining a confidence map of the radio frequency image; acquiring a camera image, inputting the camera image into an image processing network, obtaining a pre-confidence map of the camera image, calculating the correlation between the detection targets in the pre-confidence map, and obtaining a confidence map of the camera image; fusing the confidence map of the radar signal with the confidence map of the camera image to obtain a three-dimensional target detection result. The method makes full use of the semantic information of the millimeter wave radar signal, overcomes the problem of camera and laser radar failure that may be faced in the target detection process, makes the target detection have high robustness, and can perform target detection in harsh and complex environments.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular relates to a radar three-dimensional target detection method and device based on multi-sensor fusion. Background Art

[0002] Millimeter-wave radar is a common driver assistance sensor today. It can maintain its stability in most driving scenarios, including different weather and lighting conditions, which makes it more robust than other sensors such as cameras and lidar. Many autonomous or assisted driving solutions focus on sensor fusion to improve accuracy and reliability.

[0003] In current research, most sensor fusion methods only utilize the positioning information in millimeter-wave radar signals, but do not effectively utilize the rich semantic information. Therefore, millimeter-wave radar can only be used as a supplement to cameras or lidars. The accuracy of three-dimensional target detection needs to be further improved, especially in harsh environments. The credibility of three-dimensional target detection results cannot be guaranteed. Summary of the invention

[0004] In view of the defects existing in the above-mentioned prior art, the present invention provides a radar three-dimensional target detection method based on multi-sensor fusion in order to improve the credibility of three-dimensional target detection results in harsh environments and improve the accuracy of detection results. The method makes full use of the rich semantic information of millimeter-wave radar, overcomes the problem of camera and lidar failure that may be encountered in the target detection process, and can obtain more stable and reliable detection results than the prior art.

[0005] The technical solution proposed by the present invention is as follows:

[0006] On the one hand, the present invention discloses a radar three-dimensional target detection method based on multi-sensor fusion, comprising the following steps:

[0007] Acquire a radar signal, perform Fourier transform on the radar signal to obtain a radio frequency image of the radar signal, input the radio frequency image into an image processing network to obtain a pre-confidence map of the radio frequency image, calculate the correlation between the detection targets in the pre-confidence map, and obtain a confidence map of the radio frequency image;

[0008] Acquire a camera image, input the camera image into an image processing network, obtain a pre-confidence map of the camera image, calculate the correlation between detection targets in the pre-confidence map, and obtain a confidence map of the camera image;

[0009] The confidence map of the radar signal is fused with the confidence map of the camera image to obtain the 3D target detection result.

[0010] Furthermore, the specific process of obtaining the radio frequency image of the radar signal is as follows:

[0011] The acquired radar signal is preprocessed; the preprocessed radar signal is fast Fourier transformed to estimate the distance of the signal; the high-frequency noise of the linear frequency modulation signal is removed by a low-pass filter; the radar signal is fast Fourier transformed for the second time along different receiving antennas to estimate the azimuth of the signal; and the radio frequency image of the radar signal is obtained according to the distance and azimuth of the signal.

[0012] Furthermore, the correlation between detection targets in the calculation pre-confidence map is calculated using a non-maximum suppression method.

[0013] Furthermore, the metric in the non-maximum suppression method is target position discrimination, and the definition of the target position discrimination is as follows:

[0014]

[0015] Where d is the distance between two detection targets; s is the distance between the detection target and the radar; k class is a classification constant, representing the classification tolerance, which is determined by the average scale information of the corresponding category.

[0016] Furthermore, the specific process of fusing the confidence map of the radar signal and the confidence map of the camera image is: projecting the confidence map of the camera image onto the confidence map of the radar signal, and calculating the fusion confidence of the detection target in the confidence map.

[0017] Furthermore, the projection relationship between the confidence map of the camera image and the confidence map of the radar signal is defined as follows:

[0018]

[0019]

[0020] Among them, (ρ c ,θ c ) represents the projection position in radar range-azimuth coordinates; (x c , z c ) represents the target position in the camera's bird's-eye view coordinates; (x or , z or ) represents the position of the radar origin in the camera bird's eye view coordinates aligned from the sensor system calibration.

[0021] Furthermore, the definition of the fusion confidence of the detection target in the calculation confidence map is as follows:

[0022]

[0023] Among them, z i is the confidence of the detected target in the radar signal confidence map, p iis the confidence of the detected target in the camera image confidence map, c is the fusion confidence of the detected target after fusing the confidence of the radar signal and the confidence of the camera image, and β is the threshold set according to the detection environment.

[0024] Furthermore, the threshold β determines the degree of trust of the camera in different detection environments.

[0025] On the other hand, the present invention discloses a radar three-dimensional target detection device based on multi-sensor fusion, comprising the following modules:

[0026] Radar signal processing module: used to acquire radar signals, perform Fourier transform on the radar signals to obtain radio frequency images of the radar signals, input the radio frequency images into an image processing network to obtain a pre-confidence map of the radio frequency images, calculate the correlation between the detection targets in the pre-confidence map, and obtain a confidence map of the radio frequency images;

[0027] A camera image processing module is used to obtain a camera image, input the camera image into an image processing network, obtain a pre-confidence map of the camera image, calculate the correlation between the detection targets in the pre-confidence map, and obtain a confidence map of the camera image;

[0028] Image fusion module: used to fuse the confidence map of the radar signal with the confidence map of the camera image to obtain the three-dimensional target detection result.

[0029] The radar three-dimensional target detection method based on multi-sensor fusion proposed in the present invention makes full use of the semantic information of the millimeter-wave radar signal compared with the existing method that only uses the positioning information in the millimeter-wave radar signal, and uses the semantic information as the main input information for target detection, instead of using the millimeter-wave radar as a supplement to the camera or lidar. This overcomes the problem of camera and lidar failure that may be encountered during the target detection process, makes the target detection more robust, and can perform target detection in harsh and complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0031] Figure 1 This is a schematic diagram of a processing process of a radar three-dimensional target detection method based on multi-sensor fusion applied to autonomous driving according to Embodiment 1 of the present invention;

[0032] Figure 2 It is a schematic diagram of a process of converting a radar signal into a radio frequency image in a radar three-dimensional target detection method based on multi-sensor fusion according to Embodiment 1 of the present invention;

[0033] Figure 3 It is a structural schematic diagram of an image processing network in a radar three-dimensional target detection method based on multi-sensor fusion according to Embodiment 1 of the present invention;

[0034] Figure 4 It is a structural diagram of a radar three-dimensional target detection device based on multi-sensor fusion according to Example 2 of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention, and the solution of the present invention will be better described through embodiments.

[0036] Example 1

[0037] Reference Figure 1 This embodiment provides a radar three-dimensional target detection method based on multi-sensor fusion, and the forwarding method includes the following steps:

[0038] S1: collects radar signals and converts them into RF images;

[0039] Specifically, refer to Figure 2 The radar model used in this embodiment is FMCW radar, which can transmit continuous frequency modulation signals and receive reflected echoes from obstacles;

[0040] S1.1: Collect the echo signal received by the radar and perform preprocessing;

[0041] S1.2: Perform fast Fourier transform (FFT) on the pre-processed signal to estimate the distance of the reflected signal.

[0042] S1.3: Use a low-pass filter (LPF) to remove high-frequency noise from all linear frequency modulation signals in each frame at a speed of 30FPS;

[0043] S1.4: Perform a second FFT on the signal along different receiving antennas to estimate the azimuth of the reflected signal, and obtain the final RF image of the radar signal based on the distance and azimuth of the reflected signal;

[0044] S2: Inputting the RF image of the radar signal into the image processing network to obtain a preset confidence map of the RF image;

[0045] Specifically, refer to Figure 3 ,The network for processing the RF image of the radar signal is composed of a 3D convolutional neural network with an autoencoder;

[0046] The radar signal RF image data set obtained in step S1 is divided into an image processing network training set, a validation set, and a test set according to the ratio of 70%, 15%, and 15%;

[0047] The input of the network is of size (CRF ,T,n,H,W) radar radio frequency image fragment R, where C RF is the number of channels in each complex RF image frame, T is the number of RF image frames in the segment, n is the number of linear frequency modulations per frame, H and W are the height and width of the RF image, respectively;

[0048] The specific structure of the network is as follows: the network is composed of an autoencoder, which is composed of an encoder and a decoder. The encoder is composed of 6 3D convolutional layers and activation layers. Each convolutional layer is followed by batch normalization (BatchNormalization), and finally connected to the Leaky-relu activation function layer. The first and second layers are composed of 64 filters with a convolution kernel size of 9*5*5, and the padding size is 4*2*2. However, the first layer uses sliding convolution with a sliding step of 1 to extract features, while the second layer uses sliding convolution with a sliding step of 2 to extract features. The third and fourth layers are both composed of 128 filters with a convolution kernel size of 9*5*5, and the padding size is 4*2*2. Similarly, the third layer uses sliding convolution with a sliding step of 1 to extract features, while the fourth layer uses sliding convolution with a sliding step of 2 to extract features. The fifth and sixth layers are both composed of 256 filters with a convolution kernel size of 9*5*5 and a padding size of 4*2*2. However, the fifth layer uses sliding convolution with a sliding step of 1 to extract features, while the sixth layer uses sliding convolution with a sliding step of 1*2*2 to extract features.

[0049] The decoder consists of 3 3D deconvolution layers and activation layers. The first layer consists of 128 filters with a convolution kernel size of 4*6*6, a padding size of 1*2*2, and a sliding deconvolution with a sliding step of 2 to restore features; the second layer consists of 64 filters with a convolution kernel size of 4*6*6, a padding size of 1*2*2, and a sliding deconvolution with a sliding step of 2 to restore features; the third layer consists of n_class filters with a convolution kernel size of 4*6*6, a padding size of 1*2*2, and a sliding deconvolution with a sliding step of 1*2*2 to restore features, where n_class is the number of types of detected targets;

[0050] Reference Figure 1 ,The output of the network is the pre-confidence map of the RF segment, and the information in the confidence map includes the distance, category, and size information of the detected target.

[0051] The loss function when training the network is binary cross entropy.

[0052] S3: calculating the correlation of the detected target in the pre-confidence map of the radio frequency image of the radar signal to obtain the confidence map of the radio frequency image;

[0053] Specifically, a non-maximum suppression method (NMS) is used, which is often used in image-based object detection to remove redundant bounding boxes from the detection results. NMS uses the intersection of intersection over union (IOU) as an evaluation criterion to determine whether to remove detection candidates that overlap too much with the highest confidence. However, since there is no bounding box definition in the RF image, the present invention defines a new metric similar to the intersection over union, called target position discrimination (TPD), which describes the correlation between two detection targets from the perspective of distance, category and size information on the pre-confidence map. The definition of TPD is as follows:

[0054]

[0055] Where d is the distance between two detection targets, in meters; s is the distance between the detection target and the radar; k class is a classification constant, representing the classification tolerance, which is determined by the average scale information of the corresponding category.

[0056] After defining TPD, the present invention adopts the position-based non-maximum suppression method for post-processing, and the steps are as follows:

[0057] (1) Use a 5×5 window to obtain all 24 adjacent peaks in all target category channels on the confidence map as the peak set

[0058] (2) Select the peak value z with the highest confidence * ∈Z, put it into the final peak set Z * , and remove it from the original Z. At the same time, each remaining peak z is calculated i TPD value;

[0059] (3) If z * With z i If the TPD value is greater than 0.75, z i Remove from the original peak set Z;

[0060] (4) Repeat steps (2) and (3) until the peak set Z becomes an empty set, and obtain the confidence map of the final radar signal RF image, that is, the target detection result of the radar signal.

[0061] S4: Obtain a camera image and input it into an image processing network to obtain a preset confidence map of the camera image;

[0062] Specifically, refer to Figure 3 ,The network that processes the camera images consists of a 2D convolutional neural network with an autoencoder;

[0063] The input of the network is a camera image of size (H, W, k), where H is the height of the camera image, W is the width of the camera image, and k is the number of channels of the image. The size of the camera image in this embodiment is (416, 416, 3);

[0064] The specific structure of the network is as follows: the network consists of an autoencoder, which consists of an encoder and a decoder. The encoder consists of 6 convolutional layers and activation layers, which is similar to the network used to process radar signal RF images, but uses a 2D convolution kernel. Each convolution layer is connected to batch normalization, and finally connected to the leaky-relu activation function layer. The first and second layers are both composed of 64 filters with a convolution kernel size of 3*3, and the padding size is 2*2, but the first layer extracts features by sliding convolution with a sliding step of 1, while the second layer extracts features by sliding convolution with a sliding step of 2. The third and fourth layers are both composed of 128 filters with a convolution kernel size of 3*3, and the padding size is 2*2. Similarly, the third layer extracts features by sliding convolution with a sliding step of 1, while the fourth layer extracts features by sliding convolution with a sliding step of 2. The fifth and sixth layers are both composed of 256 filters with a convolution kernel size of 3*3 and a padding size of 2*2. However, the fifth layer uses sliding convolution with a sliding step of 1 to extract features, while the sixth layer uses sliding convolution with a sliding step of 2*2 to extract features.

[0065] The decoder consists of 3 deconvolution layers and activation layers. The first layer consists of 128 filters with a convolution kernel size of 5*5, with a padding size of 2*2, and a sliding deconvolution with a sliding step of 2 to restore features; the second layer consists of 64 filters with a convolution kernel size of 5*5, with a padding size of 2*2, and a sliding deconvolution with a sliding step of 2 to restore features; the third layer consists of n_class filters with a convolution kernel size of 5*5, with a padding size of 2*2, and a sliding deconvolution with a sliding step of 2*2 to restore features, where n_class is the number of types of detected targets;

[0066] Reference Figure 1 ,The output of the network is a pre-confidence map of the camera image. The information in the confidence map includes the distance, category, and size information of the detected target.

[0067] The loss function when training the network is binary cross entropy.

[0068] S5: calculating the correlation of the detected target in the pre-confidence map of the camera image to obtain the confidence map of the camera image;

[0069] Specifically, the same method is used as in step S3.

[0070] S6: Fusing the confidence map of the radar signal and the confidence map of the camera image to obtain the 3D target detection result;

[0071] Specifically, the image confidence map is first projected onto the radar signal confidence map according to the following formula:

[0072]

[0073]

[0074] Among them, (ρ c ,θ c ) represents the projection position in radar range-azimuth coordinates; (x c , z c ) represents the target position in the camera's bird's-eye view coordinates; (x or , z or ) represents the position of the radar origin in the camera bird's eye view coordinates aligned from the sensor system calibration.

[0075] Then, from the perspective of the advantages of radar in position detection and image semantics, a new fusion method of radar RF image confidence map and camera image confidence map is defined. The fusion of the two confidence maps is achieved by calculating the fusion confidence of the detected targets in the two confidence maps. The definition of the fusion confidence is as follows:

[0076]

[0077] Among them, z i is the confidence level of the detected target radar signal, p i is the confidence of the target camera image, c is the fusion confidence after the confidence of the radar signal and the confidence of the camera image are fused, and β is the threshold set according to the scene. The larger the threshold β is, the less confidence is placed on the camera image. Therefore, β determines the degree of trust in the camera in different driving scenarios. Because the present invention is more inclined to be applied to harsh driving conditions, β is empirically set to 0.85.

[0078] S7: After training, perform cross-validation and testing on the image processing model to ensure accuracy and recall. Finally, observe the convergence of the loss function in the visualization window and optimize the image processing network structure in a timely manner.

[0079] Example 2

[0080] The present invention also provides a radar three-dimensional target detection device based on multi-sensor fusion, referring to Figure 4 , the device comprises the following modules:

[0081] Radar signal processing module 201: used to acquire a radar signal, perform Fourier transform on the radar signal to obtain a radio frequency image of the radar signal, input the radio frequency image into an image processing network to obtain a pre-confidence map of the radio frequency image, calculate the correlation between the detection targets in the pre-confidence map, and obtain a confidence map of the radio frequency image;

[0082] The camera image processing module 202 is used to obtain a camera image, input the camera image into an image processing network, obtain a pre-confidence map of the camera image, calculate the correlation between the detection targets in the pre-confidence map, and obtain a confidence map of the camera image;

[0083] Image fusion module 203: used to fuse the confidence map of the radar signal and the confidence map of the camera image to obtain a three-dimensional target detection result.

[0084] In addition, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, which, when executed, includes part or all of the steps of any radar three-dimensional target detection method based on multi-sensor fusion recorded in the above method embodiment.

[0085] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0087] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0088] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation methods of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

Claims

1. A radar three-dimensional target detection method based on multi-sensor fusion, characterized in that: The following steps are involved: Acquire a radar signal, perform Fourier transform on the radar signal to obtain a radio frequency image of the radar signal, input the radio frequency image into an image processing network to obtain a pre-confidence map of the radio frequency image, calculate the correlation between the detection targets in the pre-confidence map, and obtain a confidence map of the radio frequency image; Acquire a camera image, input the camera image into an image processing network, obtain a pre-confidence map of the camera image, calculate the correlation between detection targets in the pre-confidence map, and obtain a confidence map of the camera image; The confidence map of the radar signal is integrated with the confidence map of the camera image to obtain the 3D target detection result. The non-maximum suppression method is used to calculate the correlation between the detection targets in the pre-confidence map; The metric in the non-maximum suppression method is the target position discrimination, and the definition of the target position discrimination is as follows: Where d is the distance between two detection targets; s is the distance between the detection target and the radar; k class is a classification constant, representing the classification tolerance, which is determined by the average scale information of the corresponding category; The method is characterized in that the specific process of fusing the confidence map of the radar signal and the confidence map of the camera image is as follows: projecting the confidence map of the camera image onto the confidence map of the radar signal, and calculating the fusion confidence of the detection target in the confidence map; The projection relationship between the confidence map of the camera image and the confidence map of the radar signal is defined as follows: Among them, (ρ c ,θ c ) represents the projection position in radar range-azimuth coordinates; (x c , z c ) represents the target position in the camera's bird's-eye view coordinates; (x or , z or ) represents the position of the radar origin in the camera bird's eye view coordinates aligned from the sensor system calibration; It is characterized in that the definition of the fusion confidence of the detection target in the calculation confidence map is as follows: Among them, z i is the confidence of the detected target in the radar signal confidence map, p i is the confidence of the detected target in the camera image confidence map, c is the fusion confidence of the detected target after fusing the confidence of the radar signal and the confidence of the camera image, and β is the threshold set according to the detection environment.

2. The radar three-dimensional target detection method based on multi-sensor fusion according to claim 1 is characterized in that: The specific process of obtaining the radio frequency image of the radar signal is as follows: The acquired radar signal is preprocessed; the preprocessed radar signal is fast Fourier transformed to estimate the distance of the signal; the high-frequency noise of the linear frequency modulation signal is removed by a low-pass filter; the radar signal is fast Fourier transformed for the second time along different receiving antennas to estimate the azimuth of the signal; and the radio frequency image of the radar signal is obtained according to the distance and azimuth of the signal.

3. The radar three-dimensional target detection method based on multi-sensor fusion according to claim 2 is characterized in that: The threshold β determines the degree of trust of the camera in different detection environments.

4. A radar three-dimensional target detection device based on multi-sensor fusion implemented by the method according to any one of claims 1 to 3, characterized in that: Includes the following modules: Radar signal processing module: used to acquire radar signals, perform Fourier transform on the radar signals to obtain radio frequency images of the radar signals, input the radio frequency images into an image processing network to obtain a pre-confidence map of the radio frequency images, calculate the correlation between the detection targets in the pre-confidence map, and obtain a confidence map of the radio frequency images; A camera image processing module is used to obtain a camera image, input the camera image into an image processing network, obtain a pre-confidence map of the camera image, calculate the correlation between the detection targets in the pre-confidence map, and obtain a confidence map of the camera image; Image fusion module: used to fuse the confidence map of the radar signal with the confidence map of the camera image to obtain the three-dimensional target detection result.

5. A computer-readable storage medium, characterized in that: The storage medium contains the radar three-dimensional target detection method based on multi-sensor fusion as described in any one of claims 1 to 3.

Images