Radar and camera mixed level fusion target detection method based on cross-modal bidirectional interaction

Abstract

The invention provides a radar and camera mixed level fusion target detection method based on cross-modal bidirectional interaction. The method comprises the steps of performing early fusion on a radar point cloud and a camera image; extracting features of the camera branches and performing preliminary detection, and further associating radar points with the preliminarily detected targets by adopting column expansion and view cone association technologies to generate features of radar branches; bidirectional interactive fusion of a radar-assisted camera and a camera-assisted radar is carried out on the extracted features of the two branches, and cross-modal fusion features are generated; and performing secondary detection based on the cross-modal fusion features, decoding the primary detection result and the secondary detection result, and finally completing target detection. Through integration of fusion of different stages, on one hand, original data features of the radar and the camera are reserved to the maximum extent, on the other hand, detail features of the radar and the camera are captured more comprehensively through intermediate-stage fusion, bidirectional interaction of the radar and the camera is achieved, and therefore the performance and precision of target detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and more specifically, to a radar and camera hybrid-level fusion target detection method based on cross-modal bidirectional interaction. Background Art

[0002] One of the core technologies for autonomous driving is object detection, which relies on high-precision sensors to perceive the vehicle's surroundings. This detection method enables a comprehensive understanding and analysis of surrounding objects, providing essential technical support for subsequent critical tasks such as target tracking and path planning. Commonly used sensors in autonomous driving systems include radar, cameras, and lidar. Radar operates efficiently in a wide range of weather conditions and has significant advantages in ranging and speed measurement, but it lacks the ability to perceive texture information. While cameras excel at capturing detailed image information of targets, their performance degrades significantly in extreme weather conditions (such as heavy rain or haze) and in low light conditions. Furthermore, cameras have limitations in detecting the velocity of rapidly moving objects. While lidar offers strong three-dimensional perception capabilities, the large amount of data required leads to high computational overhead and high cost, making it unsuitable for practical engineering applications. Therefore, relying solely on a single sensor type will not meet the high-precision and high-safety requirements for autonomous driving.

[0003] To address these issues and address the need for cost control in engineering applications, researchers have proposed a multimodal fusion target detection method that combines cameras and radar. This method leverages the complementary performance of radar and cameras, both of which are inexpensive sensors, thereby enhancing target detection capabilities in various complex environments.

[0004] For example, fusing radar and camera data can provide more stable and reliable target recognition in the face of occlusion and lighting changes. Furthermore, this fusion method demonstrates strong anti-interference capabilities in adverse weather conditions, enabling autonomous driving systems to operate safely in a wider range of scenarios. However, most current multi-sensor feature fusion methods still tend to employ a single fusion strategy, such as early-level fusion (EF), also known as data-level fusion, middle-level fusion (MF), also known as feature-level fusion, and late-level fusion (LF), also known as decision-level fusion. Early fusion maximizes the preservation of original information, but is computationally intensive, exhibits poor fault tolerance, and presents significant fusion challenges. Middle-level fusion, however, can design appropriate feature extractors tailored to the specific needs of heterogeneous sensor data, offering lower computational overhead than early fusion. However, middle-level fusion performs poorly in extreme weather conditions such as rain, haze, or strong sunlight. This is because the camera sensor's image acquisition capabilities are often severely impacted, preventing it from effectively capturing target features, thereby reducing overall detection reliability. In late fusion, each modality independently completes feature extraction and preliminary decision-making. The system has strong scalability and can flexibly replace a single modality processing module. However, it cannot utilize complementary features between modalities, and it is necessary to establish a mapping relationship between the detection results of different sensors. When there are large differences in target detection results between different sensors, the fusion effect is greatly affected. In addition, decision-level fusion will lose a large amount of original data information.

[0005] Regardless of the aforementioned fusion method, existing approaches often overlook the potential of hybrid-level fusion (or multi-level fusion) to improve object detection performance, especially when integrating heterogeneous sensors. Furthermore, most research focuses on one-way assistance from heterogeneous sensors, such as radar-assisted cameras, without fully exploiting the complementary nature of the sensors, resulting in insufficient information exchange. Summary of the Invention

[0006] In order to overcome the defects of the above-mentioned existing multi-sensor fusion target detection technologies that mostly adopt a single fusion strategy and insufficient information interaction, the present invention provides a radar and camera hybrid-level fusion target detection method based on cross-modal bidirectional interaction. By integrating different fusion stages, namely data-level fusion and feature-level fusion, it can more comprehensively capture and analyze information in the surrounding environment, realize two-way interaction between radar and camera, thereby improving the performance and robustness of target detection, and providing more accurate environmental perception capabilities for autonomous driving systems.

[0007] In order to solve the above technical problems, the technical solutions of the present invention are as follows: A radar and camera hybrid-level fusion target detection method based on cross-modal bidirectional interaction includes the following steps: S1: Project the collected radar point cloud data onto the two-dimensional plane of the camera image, and expand the radar points into vertical lines for height completion. Then, the radar point cloud information is used as an additional channel to perform channel stitching with the image information to obtain image data with radar point cloud information. S2: performing feature extraction and multi-scale feature fusion on the image data with radar point cloud information to obtain features of the camera branch; S3: Input the features of the camera branch into a preset first regression head for prediction to obtain a preliminary detection result; S4: Expand the collected radar point cloud into columns, and then use the cone association technique to associate each radar point with each target in the preliminary detection results to obtain the characteristics of the radar branch; S5: Inputting the features of the camera branch and the features of the radar branch into a preset cross-modal feature fusion network for bidirectional interactive fusion of features to generate cross-modal fusion features; The cross-modal feature fusion network includes: a feature channel enhancement and alignment module, a dual cross attention module and an adaptive feature fusion module connected in sequence; S6: Inputting the cross-modal fusion feature into a preset second regression head for prediction to obtain a secondary detection result; S7: Decode the preliminary detection result and the secondary detection result to generate a final target detection result.

[0008] Preferably, in step S1, the radar points are projected onto vertical lines in the two-dimensional plane of the image. By projecting the points onto the vertical lines, the radar features become denser in the ordinate of the image data without losing their accuracy in the abscissa of the image.

[0009] Preferably, in step S1, channel splicing is represented as:

[0010] in, For image data with radar point cloud information, , C 、 H and W Respectively represent the number of channels, the height and width of the image; Represents the information of the camera image, including the information of the three channels of the image R, G, and B; Represents radar point cloud information, including depth, X-axis component of radial velocity, Y-axis component of radial velocity, and radar cross section information; Represents a channel splicing operation.

[0011] Preferably, in step S2, multi-scale feature extraction is performed on the image data with radar point cloud information through a pre-trained Swin Transformer network to obtain image features of four different scales; The image features of each scale are input into the corresponding multi-scale feature fusion module respectively, and the outputs of each multi-scale feature fusion module are weightedly fused as the features of the camera branch; Each of the multi-scale feature fusion modules has the same structure and includes: a 1×1 convolutional layer, an average pooling layer, a linear layer 1, a ReLU activation layer, a linear layer 2, and a Sigmoid activation layer connected in sequence; the output of the 1×1 convolutional layer is dot-producted with the output of the Sigmoid activation layer to serve as the output of the multi-scale feature fusion module.

[0012] Preferably, in step S3, the first regression head includes a 3×3 convolutional layer and a 1×1 convolutional layer connected in sequence; The preliminary detection results include the following parameters: offset, depth, heat map, width, height, dimension and rotation information of each target; In the first regression head, the focal loss function is used to predict the heatmap parameters; for the prediction of other parameters, the L1 loss function is used.

[0013] Preferably, in step S4, the cone association technology is used to associate each radar point with each target in the preliminary detection result to obtain the characteristics of the radar branch, including: For each target in the preliminary detection results, set the threshold range at the center point of the 3D bounding box of each target , associating all radar points within the threshold range with corresponding targets; wherein, Calculated according to the following formula:

[0014] in, and are the maximum and minimum values ​​of the 3D bounding box in depth respectively; The radar points associated with each target are feature extracted and mapped into a heat map, thereby generating features of the radar branch.

[0015] Preferably, in step S5, the feature channel enhancement and alignment module includes an enhancement submodule and an alignment submodule connected in sequence; The enhancement submodule includes a multi-head self-attention layer, the output of which is added to its input via a residual connection; The alignment submodule includes two branches, each including a 1×1 convolutional layer and a multi-layer perceptron; The output of the enhancement submodule is respectively input into the two branches of the alignment submodule. After weighted fusion of the outputs of the two branches of the alignment submodule, the output is used as the output of the feature channel enhancement and alignment module, which includes the aligned radar branch features and the aligned camera branch features.

[0016] Preferably, in step S5, the dual cross attention module performs DW convolution on the aligned radar branch features and the aligned camera branch features respectively to obtain the query matrix corresponding to the radar branch features: Q pc , key matrix K pc , value matrix V pc , and the query matrix corresponding to the camera branch features Q img , key matrix K img , value matrix V img ; Then camera-assisted radar feature fusion and radar-assisted camera feature fusion are performed; Among them, camera-assisted radar feature fusion includes: Q img and K pc Perform Softmax calculation and compare the result of Softmax calculation with V pc Perform matrix multiplication, perform weighted fusion of the matrix multiplication result and the aligned radar branch features, and then input them into a multi-layer perceptron to obtain the fusion features of the camera-assisted radar; Radar-assisted camera feature fusion includes: Q pc and K img Perform Softmax calculation and compare the result of Softmax calculation with V img Perform matrix multiplication, perform weighted fusion of the matrix multiplication result and the aligned camera branch features, and then input them into a multi-layer perceptron to obtain the fusion features of the radar-assisted camera.

[0017] Preferably, in step S5, the adaptive feature fusion module first uses a pooling layer and a Softmax function to process the fusion features of the camera-assisted radar and the fusion features of the radar-assisted camera respectively to obtain adaptive weights of the radar and camera branches; and then uses the adaptive weights to perform weighted fusion on the fusion features of the camera-assisted radar and the fusion features of the radar-assisted camera to obtain the cross-modal fusion features.

[0018] Preferably, in step S6, the second regression head includes a 3×3 convolutional layer and three 1×1 convolutional layers connected in sequence; The secondary detection results include the following parameters: the speed and attributes of each target, as well as the depth and rotation information of the re-optimized prediction; In the second regression head, the binary cross entropy loss function is used to predict the attribute parameters; for the prediction of other parameters, the L1 loss function is used.

[0019] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: The present invention provides a radar and camera hybrid-level fusion target detection method based on cross-modal bidirectional interaction, which comprises the following steps: early fusion of collected radar point clouds and camera images to obtain image data with radar point cloud information; multi-scale feature extraction and fusion of the image data with radar point cloud information to obtain features of the camera branch; inputting the features of the camera branch into a preset first regression head for prediction to obtain a preliminary detection result; column expansion of the collected radar point cloud is performed, and then the cone association technology is used to associate each radar point with each target in the preliminary detection result to obtain features of the radar branch; inputting the features of the camera branch and the features of the radar branch into a preset cross-modal feature fusion network for bidirectional interactive fusion of features to generate cross-modal fusion features; inputting the cross-modal fusion features into a preset second regression head for prediction to obtain a secondary detection result; finally, decoding the preliminary detection result and the secondary detection result to generate a final target detection result; The present invention combines the rich texture information captured by the camera and the precise distance information provided by the radar through multimodal three-dimensional target detection at the data-feature level. This combination method not only realizes the effective integration of the original data at an early stage, but also improves the correlation between the fused features by sharing and interactively transmitting information at the feature level. Therefore, in complex environments (such as rain, haze or strong light environments), even if the camera cannot clearly identify the target, the radar can still provide data support to ensure the effectiveness of target detection. In addition, the present invention also introduces multi-scale feature fusion and bidirectional feature interaction based on the attention mechanism, which not only improves the robustness and effectiveness of the autonomous driving system under complex conditions, but also opens up a new direction for the wider application of multimodal fusion technology in the field of autonomous driving. This innovative solution lays a solid foundation for achieving safer and more efficient autonomous driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a radar and camera hybrid-level fusion target detection method based on cross-modal bidirectional interaction provided in Example 1.

[0021] Figure 2This is a framework diagram of a radar and camera hybrid-level fusion target detection method based on cross-modal bidirectional interaction provided in Example 2.

[0022] Figure 3 This is the Swin Transformer network structure diagram provided in Example 2.

[0023] Figure 4 This is the image feature extraction framework diagram provided in Example 2.

[0024] Figure 5 This is a schematic diagram of the process of expanding the radar points provided in Example 2 into cylinders.

[0025] Figure 6 This is a two-dimensional schematic diagram of the visual cone association provided in Example 2.

[0026] Figure 7 (a) is a diagram of the cross-modal feature fusion network framework provided in Example 2; Figure 7 (b) is a structural diagram of the feature channel enhancement and alignment module provided in Example 2; Figure 7 (c) is a structural diagram of the dual cross attention module and adaptive feature fusion module provided in Example 2.

[0027] Figure 8 This is the visualization result of projecting the radar point cloud onto the two-dimensional image plane in the early fusion process provided in Example 2.

[0028] Figure 9 This is the two-dimensional target detection result in the real scene provided in Example 2.

[0029] Figure 10 This is the three-dimensional target detection result in the real scene provided in Example 2. DETAILED DESCRIPTION

[0030] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting the present application; In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size; It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0031] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0032] Example 1 like Figure 1 As shown, this embodiment provides a radar and camera hybrid level fusion target detection method based on cross-modal bidirectional interaction, including the following steps: S1: Project the collected radar point cloud data onto the two-dimensional plane of the camera image, and expand the radar points into vertical lines for height completion. Then, the radar point cloud information is used as an additional channel to perform channel stitching with the image information to obtain image data with radar point cloud information. S2: performing feature extraction and multi-scale feature fusion on the image data with radar point cloud information to obtain features of the camera branch; S3: Input the features of the camera branch into a preset first regression head for prediction to obtain a preliminary detection result; S4: Expand the collected radar point cloud into columns, and then use the cone association technique to associate each radar point with each target in the preliminary detection results to obtain the characteristics of the radar branch; S5: Inputting the features of the camera branch and the features of the radar branch into a preset cross-modal feature fusion network for bidirectional interactive fusion of features to generate cross-modal fusion features; The cross-modal feature fusion network includes: a feature channel enhancement and alignment module, a dual cross attention module and an adaptive feature fusion module connected in sequence; S6: Inputting the cross-modal fusion feature into a preset second regression head for prediction to obtain a secondary detection result; S7: Decode the preliminary detection result and the secondary detection result to generate a final target detection result.

[0033] In the specific implementation process, the data basis of this method includes two modalities, namely, camera and radar inputs; First, this method fuses camera and radar data at an early stage. Specifically, the radar point cloud data is projected onto the two-dimensional image plane using the parameter matrix of the coordinate transformation system. During this process, the radar points are expanded into vertical lines. The depth, X-axis component of the radial velocity, the Y-axis component of the radial velocity, and the radar cross section (RCS) of the target from the radar point cloud data are then channel-joined with the RGB image, completing the fusion of radar and camera data at an early stage. Then, based on the early fusion information, multi-scale feature extraction is performed on the image branch of the camera, and the extracted multi-scale features are fused to obtain the features of the camera branch; Then, the features of the camera branch are input into the preset first regression head for prediction to obtain the preliminary detection results; Then the radar point cloud is column-expanded, and the frustum correlation technique is used to obtain the features of the radar branches; Next, intermediate-level fusion is performed. This method achieves bidirectional interactive assistance and fusion of camera and radar through feature enhancement and alignment, bidirectional interaction, and adaptive feature fusion to generate cross-modal fusion features. Finally, the cross-modal fusion features are used for secondary prediction, the preliminary detection results and the secondary detection results are decoded to generate the final target detection results, thereby achieving comprehensive and high-precision prediction of the target's three-dimensional information; By integrating different fusion stages, namely data-level fusion and feature-level fusion, this method can more comprehensively capture and analyze information in the surrounding environment, realize two-way interaction between radar and camera, thereby improving the performance and robustness of target detection and providing more accurate environmental perception capabilities for autonomous driving systems.

[0034] Example 2 like Figure 2 As shown, this embodiment provides a radar and camera hybrid level fusion target detection method based on cross-modal bidirectional interaction, including the following steps: S1: Project the collected radar point cloud data onto the two-dimensional plane of the camera image, and expand the radar points into vertical lines for height completion. Then, the radar point cloud information is used as an additional channel to perform channel stitching with the image information to obtain image data with radar point cloud information. S2: performing feature extraction and multi-scale feature fusion on the image data with radar point cloud information to obtain features of the camera branch; S3: Input the features of the camera branch into a preset first regression head for prediction to obtain a preliminary detection result; S4: Expand the collected radar point cloud into columns, and then use the cone association technique to associate each radar point with each target in the preliminary detection results to obtain the characteristics of the radar branch; S5: Inputting the features of the camera branch and the features of the radar branch into a preset cross-modal feature fusion network for bidirectional interactive fusion of features to generate cross-modal fusion features; The cross-modal feature fusion network includes: a feature channel enhancement and alignment module, a dual cross attention module and an adaptive feature fusion module connected in sequence; S6: Inputting the cross-modal fusion feature into a preset second regression head for prediction to obtain a secondary detection result; S7: Decoding the preliminary detection result and the secondary detection result to generate a final target detection result; In step S1, the radar points are projected onto vertical lines in the two-dimensional image plane. By projecting the points onto the vertical lines, the radar features become denser in the vertical coordinate of the image data without losing their accuracy in the horizontal coordinate of the image. In step S1, channel splicing is represented as follows:

[0035] in, For image data with radar point cloud information, , C 、 H and W Respectively represent the number of channels, the height and width of the image; Represents the information of the camera image, including the information of the three channels of the image R, G, and B; Represents radar point cloud information, including depth, X-axis component of radial velocity, Y-axis component of radial velocity, and radar cross section information; Represents a channel splicing operation; In step S2, multi-scale feature extraction is performed on the image data with radar point cloud information using a pre-trained Swin Transformer network to obtain image features at four different scales; The image features of each scale are input into the corresponding multi-scale feature fusion module respectively, and the outputs of each multi-scale feature fusion module are weightedly fused as the features of the camera branch; Each of the multi-scale feature fusion modules has the same structure, and includes: a 1×1 convolutional layer, an average pooling layer, a linear layer 1, a ReLU activation layer, a linear layer 2, and a Sigmoid activation layer connected in sequence; the output of the 1×1 convolutional layer is dot-producted with the output of the Sigmoid activation layer to serve as the output of the multi-scale feature fusion module; In step S3, the first regression head includes a 3×3 convolutional layer and a 1×1 convolutional layer connected in sequence; The preliminary detection results include the following parameters: offset, depth, heat map, width, height, dimension and rotation information of each target; In the first regression head, the focal loss function is used to predict the heatmap parameters; for the prediction of other parameters, the L1 loss function is used; In step S4, the cone association technique is used to associate each radar point with each target in the preliminary detection result to obtain the characteristics of the radar branch, including: For each target in the preliminary detection results, set the threshold range at the center point of the 3D bounding box of each target , associating all radar points within the threshold range with corresponding targets; wherein, Calculated according to the following formula:

[0036] in, and are the maximum and minimum values ​​of the 3D bounding box in depth respectively; Extracting features from radar points associated with each target and mapping them into a heat map to generate features of the radar branch; In step S5, the feature channel enhancement and alignment module includes an enhancement submodule and an alignment submodule connected in sequence; The enhancement submodule includes a multi-head self-attention layer, the output of which is added to its input via a residual connection; The alignment submodule includes two branches, each including a 1×1 convolutional layer and a multi-layer perceptron; The output of the enhancement submodule is respectively input into the two branches of the alignment submodule, and the output of the two branches of the alignment submodule is weightedly fused as the output of the feature channel enhancement and alignment module, which includes the aligned radar branch features and the aligned camera branch features; In step S5, the dual cross attention module performs DW convolution on the aligned radar branch features and the aligned camera branch features respectively to obtain the query matrix corresponding to the radar branch features. Q pc , key matrix K pc , value matrix V pc , and the query matrix corresponding to the camera branch features Q img , key matrix K img , value matrix V img ; Then camera-assisted radar feature fusion and radar-assisted camera feature fusion are performed; Among them, camera-assisted radar feature fusion includes: Q img and K pc Perform Softmax calculation and compare the result of Softmax calculation with V pc Perform matrix multiplication, perform weighted fusion of the matrix multiplication result and the aligned radar branch features, and then input them into a multi-layer perceptron to obtain the fusion features of the camera-assisted radar; Radar-assisted camera feature fusion includes: Q pc and K img Perform Softmax calculation and compare the result of Softmax calculation with V imgPerform matrix multiplication, perform weighted fusion of the matrix multiplication result and the aligned camera branch features, and then input them into a multi-layer perceptron to obtain the radar-assisted camera fusion features; In step S5, the adaptive feature fusion module first uses a pooling layer and a softmax function to process the fusion features of the camera-assisted radar and the fusion features of the radar-assisted camera, respectively, to obtain adaptive weights of the radar and camera branches; then uses the adaptive weights to perform weighted fusion on the fusion features of the camera-assisted radar and the fusion features of the radar-assisted camera, to obtain the cross-modal fusion features; In step S6, the second regression head includes a 3×3 convolutional layer and three 1×1 convolutional layers connected in sequence; The secondary detection results include the following parameters: the speed and attributes of each target, as well as the depth and rotation information of the re-optimized prediction; In the second regression head, the binary cross entropy loss function is used to predict the attribute parameters; for the prediction of other parameters, the L1 loss function is used.

[0037] In the specific implementation process, this method mainly includes the following stages: 1) early fusion; 2) feature extraction of the camera and radar branches, and then preliminary regression detection using the features of the camera branch. Since the radar point cloud has been projected onto the two-dimensional image plane in the early fusion, and the four-channel information of the radar has been spliced ​​with the RBG three-channel information of the image, the camera branch now contains the radar information; 3) intermediate fusion, the extracted features of the two branches are bidirectionally interactively fused to generate cross-modal fusion features and perform secondary detection. At this stage, two-way feature fusion of radar-assisted camera and camera-assisted radar is achieved; 4) obtaining 3D target detection results, and sending the results of the preliminary and secondary detection to the decoder to generate high-precision 3D target detection results. The following is a detailed description. In the early fusion stage, the collected radar point cloud is first converted from the radar coordinate system to the world coordinate system. Then, the radar point cloud is highly completed, and the radar points are expanded into vertical lines. The completed radar point cloud is then projected onto the two-dimensional plane of the camera image. The radar point cloud information is then channel-joined with the camera image information to obtain image data with radar point cloud information. Then, the four-channel information of the radar point cloud (depth, X-axis component and Y-axis component of radial velocity, and RCS) is spliced ​​with the RGB three-channel information of the camera image. The image data with radar point cloud information is expressed as:

[0038] in, For image data with radar point cloud information, , C 、H and W Respectively represent the number of channels, the height and width of the image; Information representing camera images; Represents radar point cloud information; Represents a channel splicing operation; After that, image feature extraction is performed for early fusion data of radar and image , through the multi-scale feature encoder of the image (in this embodiment, the Swin Transformer neural network model, whose structure is as follows Figure 3 Extract multi-scale features , and multi-scale feature fusion is performed. In this embodiment, four multi-scale feature fusion (MFF) modules are used to process image features of four different scales. Each MFF module has the same structure and includes the following connected in sequence: a 1×1 convolutional layer, an average pooling layer, a linear layer 1, a ReLU activation layer, a linear layer 2, and a Sigmoid activation layer; wherein, the output of the 1×1 convolutional layer is dot-producted with the output of the Sigmoid activation layer, which is then used as the output of the MFF module; The schematic diagram of the entire image feature extraction is as follows Figure 4 As shown, it is expressed as:

[0039] in, is the feature of the camera branch; The weight corresponding to each layer of features is calculated through convolution and global average pooling layers; After that, a preliminary regression prediction is performed. The first regression head used consists of a 3×3 convolutional layer and a 1×1 convolutional layer connected in sequence. The preliminary detection results include the following parameters: depth, heat map, dimension, offset, rotation information, width and height of each target; For the prediction of heat map parameters, the loss function of the first regression head is focal loss, which is:

[0040] in, Indicates the number of targets, is the ground-truth heatmap of the labeled target, and are the first and second hyperparameters of the focal loss function; For the prediction of other parameters, the loss function of the first regression head is L1 loss, and the formula is:

[0041] in, represents the true value of the sample, Represents the prediction result of the regression head; Then, radar feature extraction is performed. Using the target information predicted by the first regression head, the frustum-association method is used to associate the target with the radar points. The associated radar point cloud is converted into radar branch features using a threshold method. Specifically, if Figure 5 As shown in the figure, the collected radar point cloud is first column-extended, and then a threshold is set at the center of the three-dimensional bounding box (3DBB) of the target. ,exist All radar points within the area are associated with the target. The two-dimensional diagram of the cone association is as follows: Figure 6 As shown in the figure Represents the cone length, n is 0.2, which means increasing the cone length by 20% in the radial direction; the threshold is calculated by taking the maximum and minimum values ​​of the three-dimensional bounding box in depth :

[0042] Feature extraction is performed on the associated radar points, including depth information and radial velocity on the X and Y components. For images, the depth information provided by the radar can serve as an important complementary feature. These depth and radial velocity features are mapped into a heat map to generate the features of the radar branch. The above process can be expressed as:

[0043] in, The channel representing the feature, is the normalization factor, is the eigenvalue, and It is j The center coordinates of the target on the image, and It is j The width and height of the 2D bounding box of the object; λ is a parameter that controls the width and height of the heatmap, which is proportional to the bounding box of the target. If two targets have overlapping heatmap areas, the target with the smaller depth value is selected; Then the radar and camera features are bidirectionally fused, such as Figure 7As shown in (a), this step includes three operations: Feature Channels Enhance and Alignment (FCEA), Dual Cross Attention Fusion (DCAF), and Adaptive Features Fusion (AFF). Camera Features and radar signatures Perform channel alignment and feature enhancement, expressed as:

[0044]

[0045] in, and Respectively represent the feature channel alignment and enhancement operations of the image and point cloud, and Respectively represent the camera branch features and radar branch features after channel alignment and feature enhancement; The FCEA module structure is as follows Figure 7 As shown in (b), it mainly consists of Multi-Head Self-Attention (MHSA), 1×1 convolution, and MLP (Multi-Layer Perceptron) layers. This module aims to strengthen the global contextual association of features within each modality and enhance the consistency of feature representation between radar and camera sensors, thereby facilitating the subsequent fusion of multimodal data. The features and Perform two-way interactive auxiliary fusion, expressed as:

[0046]

[0047] in, and They represent the fusion features of camera-assisted radar and radar-assisted camera, and They represent the feature fusion operation of camera-assisted radar and radar-assisted camera respectively; The structure of DCAF module and AFF module is as follows Figure 7 As shown in (c); The DCAF module uses bidirectional feature fusion to achieve interactive fusion between radar and camera features. Specifically, for the cross-attention fusion features of the image branch, the radar point cloud query matrix is ​​used to guide the reconstruction and learning of image features, and a cross-modal weight association matrix between the point cloud and image is constructed. Similarly, for the cross-attention fusion features of the radar branch, the image query matrix is ​​used to guide the reconstruction and learning of radar features. This module can effectively capture the complementarity between modalities and achieve bidirectional interactive fusion of features between radar point clouds and camera images. In the AFF module, the pooling layer and the Softmax function are used to fuse the features of the two modalities. and Processing to obtain adaptive weight W , the formula is:

[0048] Then the features and Perform adaptive fusion to generate cross-modal fusion features , expressed as:

[0049] in, W represents adaptive weight; The AFF module can more effectively fuse cross-modal attention fusion features by calculating adaptive weights of different modalities, thereby improving the performance and accuracy of object detection; Cross-modal feature fusion The second regression head is fed into the network for secondary detection. In this embodiment, the second regression head includes a 3×3 convolutional layer and three 1×1 convolutional layers connected in sequence. The secondary detection results include the following parameters: the speed and attributes of each target, as well as the re-predicted depth and rotation information. That is, the depth and rotation information are predicted in both the first and second regressions. Finally, the preliminary detection results and the secondary detection results are decoded to obtain the final target detection results; To verify the effectiveness of this method, this embodiment also conducts experiments on the public object detection dataset nuScenes, which is a dataset with complex, diverse, and large-scale driving scenes. like Figures 8-10The following figures show the visualization results of early fusion, 2D target detection results, and 3D target detection results in two different scenarios ((a) and (b)). The experimental results show that the camera-radar cross-modal bidirectional attention fusion method proposed in this embodiment significantly improves the detection accuracy in the 3D target detection task compared with traditional methods and has good robustness, fully demonstrating the excellent performance of this method in different scenarios. This provides theoretical support and application foundation for more research on intelligent transportation systems and autonomous driving technologies. This embodiment designs a multimodal, multi-level feature fusion three-dimensional target detection method that combines data level and feature level. Compared with traditional methods that adopt a single fusion strategy, this method exhibits stronger robustness in three-dimensional target detection under extreme conditions, significantly improving the accuracy and reliability of target detection. At the same time, by introducing a multi-scale feature fusion mechanism, this method can adaptively realize the fusion of multi-scale features, improve the effective utilization of multi-scale features, and thus significantly enhance the performance of the detection model. In addition, the bidirectional auxiliary fusion mechanism based on the attention mechanism proposed in this method promotes in-depth interaction between camera and radar features and realizes dual auxiliary fusion. This approach not only efficiently improves the effectiveness of radar and camera feature fusion, but also significantly improves the target detection effect under adverse conditions.

[0050] The same or similar reference numerals correspond to the same or similar components; The terms used in the drawings to describe positional relationships are for illustrative purposes only and are not to be construed as limiting the present application. Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction, characterized in that: The following steps are involved: S1: Project the collected radar point cloud data onto the two-dimensional plane of the camera image, and expand the radar points into vertical lines for height completion. Then, the radar point cloud information is used as an additional channel to perform channel stitching with the image information to obtain image data with radar point cloud information. S2: performing feature extraction and multi-scale feature fusion on the image data with radar point cloud information to obtain features of the camera branch; S3: Input the features of the camera branch into a preset first regression head for prediction to obtain a preliminary detection result; S4: Expand the collected radar point cloud into columns, and then use the cone association technique to associate each radar point with each target in the preliminary detection results to obtain the characteristics of the radar branch; S5: Inputting the features of the camera branch and the features of the radar branch into a preset cross-modal feature fusion network for bidirectional interactive fusion of features to generate cross-modal fusion features; The cross-modal feature fusion network includes: a feature channel enhancement and alignment module, a dual cross attention module and an adaptive feature fusion module connected in sequence; S6: Inputting the cross-modal fusion feature into a preset second regression head for prediction to obtain a secondary detection result; S7: Decode the preliminary detection result and the secondary detection result to generate a final target detection result.

2. The radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction according to claim 1 is characterized in that: In step S1 , the radar points are projected onto vertical lines in the two-dimensional image plane. By projecting the points onto the vertical lines, the radar features become denser in the vertical coordinate of the image data without losing their accuracy in the horizontal coordinate of the image.

3. The radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction according to claim 1 is characterized in that: In step S1, channel splicing is represented as follows: in, For image data with radar point cloud information, , C 、 H and W Respectively represent the number of channels, the height and width of the image; Represents the information of the camera image, including the information of the three channels of the image R, G, and B; Represents radar point cloud information, including depth, X-axis component of radial velocity, Y-axis component of radial velocity, and radar cross section information; Represents a channel splicing operation.

4. The radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction according to claim 1 is characterized in that: In step S2, multi-scale feature extraction is performed on the image data with radar point cloud information using a pre-trained Swin Transformer network to obtain image features at four different scales; The image features of each scale are input into the corresponding multi-scale feature fusion module respectively, and the outputs of each multi-scale feature fusion module are weightedly fused as the features of the camera branch; Each of the multi-scale feature fusion modules has the same structure and includes: a 1×1 convolutional layer, an average pooling layer, a linear layer 1, a ReLU activation layer, a linear layer 2, and a Sigmoid activation layer connected in sequence; the output of the 1×1 convolutional layer is dot-producted with the output of the Sigmoid activation layer to serve as the output of the multi-scale feature fusion module.

5. The radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction according to claim 1 is characterized in that: In step S3, the first regression head includes a 3×3 convolutional layer and a 1×1 convolutional layer connected in sequence; The preliminary detection results include the following parameters: offset, depth, heat map, width, height, dimension and rotation information of each target; In the first regression head, the focal loss function is used to predict the heatmap parameters; For the prediction of other parameters, the L1 loss function is used.

6. The radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction according to claim 5 is characterized in that: In step S4, the cone association technique is used to associate each radar point with each target in the preliminary detection result to obtain the characteristics of the radar branch, including: For each target in the preliminary detection results, set the threshold range at the center point of the 3D bounding box of each target , associating all radar points within the threshold range with corresponding targets; wherein, Calculated according to the following formula: in, and are the maximum and minimum values ​​of the 3D bounding box in depth respectively; The radar points associated with each target are feature extracted and mapped into a heat map, thereby generating features of the radar branch.

7. The radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction according to claim 1 is characterized in that: In step S5, the feature channel enhancement and alignment module includes an enhancement submodule and an alignment submodule connected in sequence; The enhancement submodule includes a multi-head self-attention layer, the output of which is added to its input via a residual connection; The alignment submodule includes two branches, each including a 1×1 convolutional layer and a multi-layer perceptron; The output of the enhancement submodule is respectively input into the two branches of the alignment submodule. After weighted fusion of the outputs of the two branches of the alignment submodule, the output is used as the output of the feature channel enhancement and alignment module, which includes the aligned radar branch features and the aligned camera branch features.

8. The radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction according to claim 7 is characterized in that: In step S5, the dual cross attention module performs DW convolution on the aligned radar branch features and the aligned camera branch features respectively to obtain the query matrix corresponding to the radar branch features. Q pc , key matrix K pc , value matrix V pc , and the query matrix corresponding to the camera branch features Q img , key matrix K img , value matrix V img ; Then camera-assisted radar feature fusion and radar-assisted camera feature fusion are performed; Among them, camera-assisted radar feature fusion includes: Q img and K pc Perform Softmax calculation and compare the result of Softmax calculation with V pc Perform matrix multiplication, perform weighted fusion of the matrix multiplication result and the aligned radar branch features, and then input them into a multi-layer perceptron to obtain the fusion features of the camera-assisted radar; Radar-assisted camera feature fusion includes: Q pc and K img Perform Softmax calculation and compare the result of Softmax calculation with V img Perform matrix multiplication, perform weighted fusion of the matrix multiplication result and the aligned camera branch features, and then input them into a multi-layer perceptron to obtain the fusion features of the radar-assisted camera.

9. The radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction according to claim 8 is characterized in that: In step S5, the adaptive feature fusion module first uses a pooling layer and a Softmax function to process the fusion features of the camera-assisted radar and the fusion features of the radar-assisted camera, respectively, to obtain adaptive weights of the radar and camera branches; and then uses the adaptive weights to perform weighted fusion on the fusion features of the camera-assisted radar and the fusion features of the radar-assisted camera to obtain the cross-modal fusion features.

10. The radar and camera hybrid fusion target detection method based on cross-modal bidirectional interaction according to claim 1 is characterized in that: In step S6, the second regression head includes a 3×3 convolutional layer and three 1×1 convolutional layers connected in sequence; The secondary detection results include the following parameters: the speed and attributes of each target, as well as the depth and rotation information of the re-optimized prediction; In the second regression head, the binary cross entropy loss function is used to predict the attribute parameters; for the prediction of other parameters, the L1 loss function is used.

Images