A target detection method based on three-dimensional information fusion camera and millimeter wave radar

Abstract

The application is suitable for the field of intelligent driving technology, and provides a target detection method based on three-dimensional information fusion camera and millimeter wave radar, which comprises the following steps: data acquisition and preprocessing; data processing and vehicle detection; unified coordinate system; constructing attention mechanism fusion neural network model AFnet. Under the decision level fusion framework, by introducing the distance estimation method based on deep learning, the fusion feature information is expanded, and the fusion of camera and millimeter wave radar in three-dimensional space information is realized; especially under complex working conditions, three-dimensional information of detected objects can be continuously provided to the lower layer, the robustness of the target detection algorithm is stronger, and the driving safety is guaranteed. The AFnet model realizes the fusion matching of the detection targets of each sensor in the three-dimensional space level, overcomes the limitations of traditional constraint conditions, and improves the accuracy and reliability of the fusion matching and target detection algorithm.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving technology, and in particular relates to a target detection method based on a three-dimensional information fusion camera and millimeter-wave radar. Background Technology

[0002] Intelligent driving systems have become a highly anticipated field in recent years and are a significant trend in future automotive development. To improve system stability and reliability, researchers have begun exploring multi-sensor fusion technology, which fuses data from multiple sensors. Through data processing and analysis, this technology can yield more accurate environmental perception results. Millimeter-wave radar and cameras are commonly used sensors in intelligent driving systems. Currently, fusion methods based on millimeter-wave radar and cameras are mainly classified into data-level fusion, feature-level fusion, and decision-level fusion. While data-level and feature-level fusion have certain advantages, they struggle to guarantee robustness and reliability in complex road environments. Decision-level fusion frameworks, however, better meet the practical needs of intelligent driving systems. This method fuses decision results from different sensors and makes a final decision based on these results, thereby achieving more accurate and reliable environmental perception.

[0003] In complex traffic environments, vehicles experience occlusion and overlap, and their contour features change as both surrounding and vehicular vehicles move. While prediction-based tracking methods are commonly used to address this issue, they may not be suitable for certain specific scenarios. Figure 1 Without proper occlusion and overlap, predictive tracking becomes difficult to perform. In the image, the green car and the yellow car (the main vehicle) are driving side-by-side. Due to occlusion and overlap, the blue car might remain undetected by the main vehicle. In this situation, multi-frame sequential predictive tracking fails. As the vehicle approaches, by the time the sensors detect the blue car, the main vehicle's reaction time may be very short, potentially leading to a traffic accident. Therefore, improving the system's single-frame detection rate is essential. The best way to improve the detection rate based on a single frame is to combine the detection results from various sensors as much as possible. Through multi-sensor information fusion, the perception system's tolerance to occlusion and overlapping vehicles across sensors can be improved, thereby enabling earlier target detection.

[0004] For multi-sensor information fusion, common fusion methods include: probability matching, which fuses the detection results of each sensor, calculates the probability value of the detection box matching, and uses the probability value as a weight to fuse the detection results to obtain the final detection result; distance-based fusion matching, which determines whether the detection boxes of each sensor belong to the same vehicle based on the distance between them; fusion based on the intersection-over-union ratio (IOU) of the detection boxes, where if the IOU exceeds a preset threshold, the detection boxes of each sensor are considered to belong to the same target; and fusion based on the historical trajectory of each detection branch, thereby improving the accuracy of fusion matching. However, the vehicle detection data obtained by cameras and millimeter-wave radar have certain deviations, and the data formats of the detection results are also inconsistent. Different fusion matching algorithms have different requirements for data formats, which will affect the accuracy of the fusion matching algorithm. In the existing technology, most multi-sensor framework designs are master-slave relationships, which results in low redundancy and accuracy of the entire system. Moreover, for existing fusion matching algorithms, traditional fusion matching methods are based on certain constraints, which often lead to missed fusion and false fusion due to inappropriate threshold or constraint settings, seriously threatening the safety of intelligent driving. Therefore, we propose a target detection method based on a 3D information fusion camera and millimeter-wave radar. Summary of the Invention

[0005] The purpose of this invention is to provide a target detection method based on a three-dimensional information fusion camera and millimeter-wave radar, which aims to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A target detection method based on a 3D information fusion camera and millimeter-wave radar includes the following steps:

[0008] Step 1: Data Acquisition and Preprocessing: First, the raw data information acquired by the camera and millimeter-wave radar is acquired, the data is preprocessed, and the data time is synchronized; then, the preprocessed data is input into the camera detection branch and the millimeter-wave radar detection branch respectively.

[0009] Step 2, Data Processing and Vehicle Detection: In the camera detection branch, the image undergoes pixel and size processing before being input into the network; the camera detection branch performs vehicle detection and depth estimation, and extracts disparity from the depth disparity map based on the vehicle detection results to obtain the three-dimensional information of the detected vehicle; in the millimeter-wave radar detection branch, invalid point cloud data information is filtered out by velocity filtering, and clustering algorithms are used at the four-dimensional level to achieve vehicle detection and differentiation.

[0010] Step 3: Unify the coordinate system: Transform the millimeter-wave radar data to place it in the same coordinate system as the image data.

[0011] Step 4: Construct the attention mechanism fusion neural network model AFnet: Input the results of the camera detection branch and the millimeter-wave radar detection branch into the AFnet model, perform fusion matching, and obtain the final detection result.

[0012] Furthermore, the raw data acquired by the camera is RGB three-channel image data. The raw data acquired by the millimeter-wave radar is the object's azimuth angle β, elevation angle α, distance d, and relative velocity V. rel According to the vehicle's speed V self Calculate the absolute velocity V of the object. abs The polar coordinates are then converted to Cartesian coordinates using the following formula:

[0013] x=d*cosα*cosβ

[0014] y = d*cosα*sinβ

[0015] z = d*sinα

[0016] V abs =V rel +V self *cosα*cosβ

[0017] Using millimeter-wave radar as a reference, data time synchronization is performed on the camera.

[0018] Furthermore, in the camera detection branch, the RGB image of the left camera is input into the object detection algorithm, and the RGB images of both cameras are input into the depth estimation algorithm. The depth estimation algorithm uses binocular stereo matching, and the specific operation is as follows:

[0019] For any point P(X) in the image C Y C Using binocular stereo matching, corresponding feature points P in the left and right images are obtained. l (x l y l ) and P r (x r y r Then, calculate the disparity d of corresponding feature points in the left and right images. After obtaining the disparity d, calculate the depth information Z of point P according to the formula. C The formula is as follows:

[0020]

[0021] Where f is the camera focal length and B is the baseline length, which is the distance between the left and right cameras.

[0022] Furthermore, in the millimeter-wave radar detection branch, after clustering, the bounding box of each vehicle is mapped, according to the front view size of a conventional vehicle. size Based on the obtained vehicle depth information distance Mapping is performed, and the bounding box information (Pixel) of the vehicle detected by millimeter-wave radar on the image is obtained using a formula. size The formula is as follows:

[0023]

[0024] Where f is the camera focal length of the image being mapped.

[0025] Furthermore, the specific operation of step three is as follows:

[0026] Transform the coordinates of the millimeter-wave radar point cloud data into the image coordinate system of the left camera;

[0027] First, the point cloud data (X) of any millimeter-wave radar is... r V r Z r Transform to world coordinate system (X) w Y w Z w The formula is as follows:

[0028]

[0029] Where R r It is the rotation matrix from the millimeter-wave radar coordinate system to the world coordinate system, T r It is a translation matrix from the millimeter-wave radar coordinate system to the world coordinate system;

[0030] Then convert the point cloud (X) in the world coordinate system w Y w Z w Transformed to camera coordinate system (X) c Y c Z c The formula is as follows:

[0031]

[0032] Where R c T is the rotation matrix from the camera coordinate system to the world coordinate system. c It is the translation matrix from the camera coordinate system to the world coordinate system;

[0033] Finally, the point cloud in the camera coordinate system (X) c Y c Zc The transformation to (u, v) in the image coordinate system is as follows:

[0034]

[0035] Where K is the intrinsic parameter matrix of the camera;

[0036] By combining the formulas, we obtain a comprehensive expression conversion formula, which can convert any point cloud (X) under millimeter-wave radar. r Y r Z r Transforming to the image coordinate system (u, v), the comprehensive transformation formula is expressed as:

[0037]

[0038] Where T rel It is the translation matrix between the camera and the millimeter-wave radar.

[0039] Furthermore, in step four, based on the Encoder module in Transformer, an AFnet model is constructed using a multilayer perceptron (MLP). The input of the AFnet model consists of a two-dimensional matrix composed of multiple sequences. The data information is converted into natural language sequences for processing. The first five columns of data (X1, Y1, X2, Y2, Z) are the coordinates of the upper left corner (X1, Y1), the lower right corner (X2, Y2), and the depth information Z of the bounding box detected by the camera. The last three columns of data (X3, Y3, Z3) are the three-dimensional spatial coordinates of the center point of the object detected by the millimeter-wave radar. At the same time, the number of objects detected by the camera detection branch and the millimeter-wave radar detection branch are a and b, respectively. Therefore, the AFnet neural network has a total of n = a * b input sequences.

[0040] In the AFNet model, a total of four encoders are used to encode the feature information into an n*8 matrix representation. This n*8 matrix is ​​then input into the FC1 fully connected layer of the MLP, and non-linearly connected to the FC2 fully connected layer via the ReLU activation function. Finally, the matching probability value for each sequence is calculated using the Sgimoid function and the following formula:

[0041]

[0042] Where x is the value of the output of the previous hidden layer, and e is the natural constant.

[0043] Furthermore, the AFnet model is trained using binary cross-entropy as the loss function, defined as follows:

[0044]

[0045] in y is the probability that the AFnet model predicts the same vehicle, and y is the label of the sequence in the input data. If the sequence formed by the detection results of the camera and the millimeter-wave radar is the same vehicle, the value is 1, otherwise the value is 0.

[0046] For the results predicted by the AFnet model, if the probability value is greater than 0.5, it is considered that the two vehicles detected by the two sensors in the sequence are the same vehicle; otherwise, they are not.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] 1. Under the decision-level fusion framework, by introducing a distance estimation method based on deep learning, the fusion feature information is expanded, realizing the fusion of camera and millimeter-wave radar in three-dimensional spatial information; especially under complex working conditions, it can continuously provide the lower layer with three-dimensional information of detected objects, making the target detection algorithm more robust and ensuring driving safety.

[0049] 2. A deep learning-based attention mechanism fusion matching network (AFnet) is proposed to achieve fusion matching of the detected targets of each sensor in three-dimensional space, overcoming the limitations of traditional constraints and improving the accuracy and reliability of fusion matching and target detection algorithms.

[0050] 3. Based on the density (DBSCAN) clustering algorithm, velocity feature information is introduced to achieve clustering of point cloud data in four dimensions, thereby improving the accuracy and reliability of millimeter-wave radar detection. Attached Figure Description

[0051] Figure 1 This is a diagram illustrating the occlusion and overlap of vehicles in certain special scenarios.

[0052] Figure 2 This is a schematic diagram of the structure of the present invention.

[0053] Figure 3 This is a schematic diagram illustrating the principle of obtaining depth information based on parallax. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0056] like Figure 2As shown, an embodiment of the present invention provides a target detection method based on a three-dimensional information fusion camera and millimeter-wave radar, comprising the following steps:

[0057] Step 1: Data Acquisition and Preprocessing: First, the raw data information acquired by the camera and millimeter-wave radar is acquired, the data is preprocessed, and the data time is synchronized; then, the preprocessed data is input into the camera detection branch and the millimeter-wave radar detection branch respectively.

[0058] Step 2, Data Processing and Vehicle Detection: In the camera detection branch, the image undergoes pixel and size processing before being input into the network; the camera detection branch performs vehicle detection and depth estimation, and extracts disparity from the depth disparity map based on the vehicle detection results to obtain the three-dimensional information of the detected vehicle; in the millimeter-wave radar detection branch, invalid point cloud data information is filtered out by velocity filtering, and clustering algorithms are used at the four-dimensional level to achieve vehicle detection and differentiation.

[0059] Step 3: Unify the coordinate system: Transform the millimeter-wave radar data to place it in the same coordinate system as the image data.

[0060] Step 4: Construct the attention mechanism fusion neural network model AFnet: Input the results of the camera detection branch and the millimeter-wave radar detection branch into the AFnet model, perform fusion matching, and obtain the final detection result.

[0061] In this embodiment of the invention, preferably, the purpose of data processing is to reduce the impact of noise on detection accuracy. In the image detection branch, the pixel size of the image affects the accuracy of the algorithm, but excessively large pixels also increase the computational load. Therefore, it is necessary to reasonably balance the two and modify the image to meet appropriate requirements. In the millimeter-wave radar branch, point cloud data also contains noise from the ground and buildings. To improve detection accuracy and reduce computational load, these invalid data need to be filtered out. The common characteristic of these invalid noise points is that their absolute velocity is zero. Therefore, based on the absolute velocity of the point cloud obtained in the preprocessing stage, they can be effectively filtered out.

[0062] In a preferred embodiment of the present invention, the raw data acquired by the camera is RGB three-channel image data. In practical applications, if image distortion occurs, image correction is required.

[0063] In a preferred embodiment of the present invention, the raw data collected by the millimeter-wave radar includes the object's azimuth angle β, elevation angle α, distance d, and relative velocity V. rel According to the vehicle's speed V self Calculate the absolute velocity V of the object. abs The polar coordinates are then converted to Cartesian coordinates using the following formula:

[0064] x=d*cosα*cosβ

[0065] y = d*cosα*sinβ

[0066] z = d*sinα

[0067] V abs =V rel +V self *cosα*cosβ

[0068] Using millimeter-wave radar as a reference, data time synchronization is performed on the camera.

[0069] In this embodiment of the invention, preferably, the acquisition frequencies of the camera and the millimeter-wave radar are different, and it is necessary to synchronize the data information of the camera and the millimeter-wave radar in time. The principle is to use the millimeter-wave radar with a lower acquisition frequency as a reference and synchronize the data of the camera with a higher sampling frequency in time, so as to realize that the data input to the two detection branches are at the same time.

[0070] like Figure 3 As shown, in a preferred embodiment of the present invention, in the camera detection branch, the RGB image of the left camera is input into the object detection algorithm, and the RGB images of both cameras are input into the depth estimation algorithm. The depth estimation algorithm uses binocular stereo matching, and the specific operation is as follows:

[0071] based on Figure 3 The principle is that for any point P(X) in the image... C Y C Using binocular stereo matching, corresponding feature points P in the left and right images are obtained. l (x l y l ) and P r (x r y r Then, calculate the disparity d of corresponding feature points in the left and right images. After obtaining the disparity d, calculate the depth information Z of point P according to the formula. C The formula is as follows:

[0072]

[0073] Where f is the camera focal length and B is the baseline length, which is the distance between the left and right cameras.

[0074] In this embodiment of the invention, preferably, in vehicle detection, the camera and millimeter-wave radar identify the vehicle using their respective vehicle detection algorithms. In the field of image target recognition, there are one-stage and two-stage recognition algorithms. Compared to two-stage detection algorithms, after several years of research and development, one-stage algorithms, represented by YOLOv5, have significantly improved accuracy, with a single-frame detection time of less than 20 milliseconds. Considering the limited feature information of real-world objects when they are occluded or overlapping, the YOLOv5x model is chosen. This model increases the number of residual components in each module structure, giving it a deeper and wider network structure, thus increasing its network feature extraction and feature fusion capabilities, making it more suitable for the scenario requirements. For depth estimation algorithms, monocular depth estimation has a higher error than binocular depth estimation. To improve the fusion matching accuracy, a binocular stereo matching depth estimation strategy is adopted.

[0075] PSMnet is an end-to-end stereo matching method based on convolutional neural networks. This method employs a pyramid structure and dilated convolution to extract multi-level feature information, incorporating global contextual information into image features and expanding the receptive field of view. It can automatically match feature points and output the disparity of all pixels. Then, based on the center point of the detection box identified by the object detection algorithm, the disparity value is extracted, and the depth information of the object is calculated using a formula, thus obtaining the object's three-dimensional spatial position information.

[0076] In a preferred embodiment of the present invention, in the millimeter-wave radar detection branch, after clustering, the bounding box of each vehicle is mapped, according to the front view size of a conventional vehicle. size Based on the obtained vehicle depth information aistance Mapping is performed, and the bounding box information (Pixel) of the vehicle detected by millimeter-wave radar on the image is obtained using a formula. size The formula is as follows:

[0077]

[0078] Where f is the camera focal length of the image being mapped.

[0079] In this embodiment of the invention, preferably, the velocity-filtered point cloud data is differentiated, thus identifying different point clouds representing different objects. Millimeter-wave radar in intelligent driving is tasked with detecting vehicles, pedestrians, and non-motorized vehicles. The point cloud data of these detected vehicles is characterized by a clustered distribution, making density-based clustering algorithms more suitable. After preprocessing, the millimeter-wave radar point cloud data is transformed from a polar coordinate system to a Cartesian coordinate system with the vehicle itself as the origin. Combined with the absolute velocity of the detected objects, this forms four-dimensional point cloud data. In actual vehicle operation, there are speed differences between vehicles. Sometimes, vehicles that are very close together are easily clustered into the same vehicle. Therefore, this invention also incorporates speed information into the density clustering algorithm. The millimeter-wave radar object detection algorithm flow is as follows:

[0080] Input: Sample set D = {x1, x2, ..., x} m};

[0081] Domain parameters (ε, MinPts).

[0082] process:

[0083]

[0084]

[0085] Output: Processed data and an array of cluster centroids

[0086] After clustering, each vehicle's cluster is obtained, and the cluster center point and average speed are extracted as the vehicle's feature information. Simultaneously, for subsequent visualization and partial fusion algorithms, we also need to map the bounding box of each vehicle. Considering that vehicles are mostly presented in front view form, we use the standard front view size for Vehicles. size Using a formula, the bounding box information (Pixel) of the vehicle detected by millimeter-wave radar on the image is obtained. size Using the above method, the four-dimensional spatial information and mapped bounding box information of the detected vehicle are obtained. This information is then passed to the next layer for subsequent fusion and matching.

[0087] In a preferred embodiment of the present invention, the specific operation of step three is as follows:

[0088] Transform the coordinates of the millimeter-wave radar point cloud data into the image coordinate system of the left camera;

[0089] First, the point cloud data (X) of any millimeter-wave radar is... r V r Z r Transform to world coordinate system (X) w Yw Z w The formula is as follows:

[0090]

[0091] Where R r It is the rotation matrix from the millimeter-wave radar coordinate system to the world coordinate system, T r It is a translation matrix from the millimeter-wave radar coordinate system to the world coordinate system;

[0092] Then convert the point cloud (X) in the world coordinate system w Y w Z w Transformed to camera coordinate system (X) c Y c Z c The formula is as follows:

[0093]

[0094] Where R c T is the rotation matrix from the camera coordinate system to the world coordinate system. c It is the translation matrix from the camera coordinate system to the world coordinate system;

[0095] Finally, the point cloud in the camera coordinate system (X) c Y c Z c The transformation to (u, v) in the image coordinate system is as follows:

[0096]

[0097] Where K is the intrinsic parameter matrix of the camera;

[0098] By combining the formulas, we obtain a comprehensive expression conversion formula, which can convert any point cloud (X) under millimeter-wave radar. r Y r Z r Transforming to the image coordinate system (u, v), the comprehensive transformation formula is expressed as:

[0099]

[0100] Where T rel It is the translation matrix between the camera and the millimeter-wave radar.

[0101] In this embodiment of the invention, preferably, the coordinate systems of the detected vehicle position information obtained from the two detection branches are inconsistent and cannot be fused and matched. Therefore, it is necessary to transform them to the same coordinate system. For ease of observation and calculation, the coordinates of the millimeter-wave radar point cloud data are transformed to the image coordinate system of the left camera.

[0102] In a preferred embodiment of the present invention, in step four, an AFnet model is constructed based on the Encoder module in Transformer and using a multilayer perceptron (MLP). The input of the AFnet model consists of a two-dimensional matrix composed of multiple sequences. The data information is converted into natural language sequences for processing. The first five columns of data (X1, Y1, X2, Y2, Z) are the coordinates of the upper left corner (X1, Y1), the lower right corner (X2, Y2), and the depth information Z of the bounding box detected by the camera. The last three columns of data (X3, Y3, Z3) are the three-dimensional spatial coordinates of the center point of the object detected by the millimeter-wave radar. At the same time, the number of objects detected by the camera detection branch and the millimeter-wave radar detection branch are a and b, respectively. Therefore, the AFnet neural network has a total of n = a * b input sequences.

[0103] In the AFNet model, a total of four encoders are used to encode the feature information into an n*8 matrix representation. This n*8 matrix is ​​then input into the FC1 fully connected layer of the MLP, and non-linearly connected to the FC2 fully connected layer via the ReLU activation function. Finally, the matching probability value for each sequence is calculated using the Sgimoid function and the following formula:

[0104]

[0105] Where x is the value of the output of the previous hidden layer, and e is the natural constant.

[0106] In this embodiment of the invention, to better fuse the detection results of two detection branches, an attention mechanism fusion neural network (AFnet) is proposed. Based on the encoder module in the Transformer, this invention utilizes a multilayer perceptron (MLP) to construct the AFnet network model, which can effectively fuse and match vehicles detected by two sensors. The encoder consists of a self-attention mechanism and a feedforward neural network. The self-attention mechanism's operational rules consider not only the relationships between features within a sequence but also the relationships between sequences. To better utilize this self-attention mechanism, the three-dimensional spatial coordinate information obtained by each detection branch at the same time is combined into a sequence.

[0107] In addition, the present invention also uses a multi-head attention mechanism in the Encoder, which has multiple self-attention layer subspaces. Within each attention layer subspace, an attention score is calculated for each feature. After the multi-head attention mechanism, each feature will receive multiple weight scores, which are then multiplied by an additional weight matrix to finally obtain the score for each feature. This enables better weight allocation of matching information, making the fusion matching more accurate.

[0108] The AFNet model uses four encoders, each with the same structure, but their parameters differ during actual training. The output of the last encoder contains information about the sequence in the input data and its relationship with other sequences, encoding this feature information into an n*8 matrix. This n*8 matrix is ​​then fed into the FC1 fully connected layer in the MLP to better extract feature information from the sequence. A ReLU activation function is used to non-linearly connect it to the FC2 fully connected layer, which acts as the output layer and extracts the feature information from each sequence using weighted summation. The FC1 fully connected layer contains 64 hidden neurons, while the FC2 fully connected layer contains one neuron. Finally, the Sgimoid function is used to calculate the matching (0, 1) probability value for each sequence using a formula.

[0109] In a preferred embodiment of the present invention, the AFnet model is trained using binary cross-entropy as the loss function, as defined below:

[0110]

[0111] in y is the probability that the AFnet model predicts the same vehicle, and y is the label of the sequence in the input data. If the sequence formed by the detection results of the camera and the millimeter-wave radar is the same vehicle, the value is 1, otherwise the value is 0.

[0112] For the results predicted by the AFnet model, if the probability value is greater than 0.5, it is considered that the two vehicles detected by the two sensors in the sequence are the same vehicle; otherwise, they are not.

[0113] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A target detection method based on a 3D information fusion camera and millimeter-wave radar, characterized in that, Includes the following steps: Step 1: Data Acquisition and Preprocessing: First, the raw data information acquired by the camera and millimeter-wave radar is acquired, the data is preprocessed, and data time synchronization is performed. The preprocessed data is then input into the camera detection branch and the millimeter-wave radar detection branch, respectively. Step 2, Data Processing and Vehicle Detection: In the camera detection branch, the image undergoes pixel and size processing before being input into the network; The camera detection branch performs vehicle detection and depth estimation. Based on the vehicle detection results, it extracts the disparity from the depth disparity map to obtain the three-dimensional information of the detected vehicle. In the millimeter-wave radar detection branch, invalid point cloud data information is filtered out by velocity filtering, and clustering algorithms are used at the four-dimensional level to achieve vehicle detection and differentiation; Step 3: Unify the coordinate system: Transform the millimeter-wave radar data to place it in the same coordinate system as the image data. Step 4: Construct the attention mechanism fusion neural network model AFnet: Input the results of the camera detection branch and the millimeter-wave radar detection branch into the AFnet model, perform fusion matching, and obtain the final detection result; In step four, based on the Encoder module in Transformer, an AFnet model is constructed using a multilayer perceptron (MLP). The input of the AFnet model consists of a two-dimensional matrix composed of multiple sequences. The data information is converted into natural language sequences for processing. The first five columns of data (X1, Y1, X2, Y2, Z) are the coordinates of the upper left corner (X1, Y1), the lower right corner (X2, Y2), and the depth information Z of the bounding box detected by the camera. The last three columns of data (X3, Y3, Z3) are the three-dimensional spatial coordinates of the center point of the object detected by the millimeter-wave radar. At the same time, the number of objects detected by the camera detection branch and the millimeter-wave radar detection branch are a and b, respectively. Therefore, the AFnet neural network has a total of n = a * b input sequences. In the AFNet model, a total of four encoders are used to encode the feature information into an n*8 matrix representation. This n*8 matrix is ​​then input into the FC1 fully connected layer of the MLP, and non-linearly connected to the FC2 fully connected layer via the ReLU activation function. Finally, the matching probability value for each sequence is calculated using the Sgimoid function and the following formula: ； Where x is the value of the output of the previous hidden layer, and e is the natural constant.

2. The target detection method based on a three-dimensional information fusion camera and millimeter-wave radar according to claim 1, characterized in that, The raw data collected by the millimeter-wave radar is the azimuth angle of the object. Altitude angle ,distance relative speed According to the vehicle's speed Calculate the absolute velocity of the object. The polar coordinates are then converted to Cartesian coordinates using the following formula: ； ； ； ； Using millimeter-wave radar as a reference, data time synchronization is performed on the camera.

3. The target detection method based on a three-dimensional information fusion camera and millimeter-wave radar according to claim 1, characterized in that, In the camera detection branch, the RGB image of the left camera is input into the object detection algorithm, and the RGB images of both cameras are input into the depth estimation algorithm. The depth estimation algorithm uses binocular stereo matching, and the specific operation is as follows: For any point in the image Binocular stereo matching is used to obtain corresponding feature points in the left and right images. and Then, calculate the disparity d of corresponding feature points in the left and right images. After obtaining the disparity d, calculate the point value according to the formula. depth information The formula is as follows: ； Where f is the camera focal length and B is the baseline length, which is the distance between the left and right cameras.

4. The target detection method based on a three-dimensional information fusion camera and millimeter-wave radar according to claim 1, characterized in that, In the millimeter-wave radar detection branch, after clustering, the bounding box of each vehicle is mapped, according to the size of a conventional vehicle's front view. Based on the obtained vehicle depth information Mapping is performed, and the bounding box information of the vehicle detected by millimeter-wave radar on the image is obtained using a formula. The formula is as follows: ； Where f is the camera focal length of the image being mapped.

5. The target detection method based on a three-dimensional information fusion camera and millimeter-wave radar according to claim 1, characterized in that, The specific operation of step three is as follows: Transform the coordinates of the millimeter-wave radar point cloud data into the image coordinate system of the left camera; First, take any point cloud data from the millimeter-wave radar. Transform to world coordinate system The formula is as follows: ； in It is a rotation matrix from the millimeter-wave radar coordinate system to the world coordinate system. It is a translation matrix from the millimeter-wave radar coordinate system to the world coordinate system; Then convert the point cloud in the world coordinate system Transformed to camera coordinate system The formula is as follows: ； in It is the rotation matrix from the camera coordinate system to the world coordinate system. It is the translation matrix from the camera coordinate system to the world coordinate system; Finally, the point cloud is plotted in the camera coordinate system. Transformed to image coordinate system The formula is as follows: ； Where K is the intrinsic parameter matrix of the camera; By combining the formulas, we obtain a comprehensive expression conversion formula, which can convert any point cloud under millimeter-wave radar. Transform to image coordinate system The comprehensive expression conversion formula is as follows: ； in It is the translation matrix between the camera and the millimeter-wave radar.

6. The target detection method based on a three-dimensional information fusion camera and millimeter-wave radar according to claim 1, characterized in that, The AFnet model is trained using binary cross-entropy as the loss function, defined as follows: ； in y is the probability that the AFnet model predicts the same vehicle, and y is the label of the sequence in the input data. If the sequence formed by the detection results of the camera and the millimeter-wave radar is the same vehicle, the value is 1, otherwise the value is 0. For the results predicted by the AFnet model, if the probability value is greater than 0.5, it is considered that the two vehicles detected by the two sensors in the sequence are the same vehicle; otherwise, they are not.

Images