Vehicle detection method based on millimeter wave radar and vision fusion

Abstract

The invention relates to a vehicle detection method based on millimeter wave radar and vision fusion, and belongs to the technical field of intelligent traffic and multi-mode perception. The method is realized through the following steps: firstly, obtaining sparse point cloud data of a millimeter wave radar, extracting a vehicle target point cloud by adopting a DBSCAN clustering algorithm, and further generating a single-channel binary mask graph R0; then, a double-input fusion network is constructed, the network can process the radar mask image and the RGB image at the same time, and cross fusion of multi-scale features is achieved through an R-PAN structure; finally, an improved double cross attention mechanism (DCA) is introduced, and the DCA comprises dynamic weight channel attention (DW-CCA), deformable space attention (DF-SCA) and cross-modal consistency constraint (CMC), so that deep enhancement of fusion features is realized. According to the invention, the precision and multi-modal robustness of vehicle detection in a complex scene are significantly improved, and an efficient and reliable detection scheme is provided for the field of intelligent traffic.

Description

Technical Field

[0001] This invention belongs to the field of multimodal perception technology in intelligent transportation systems, specifically relating to a vehicle target detection method that integrates millimeter-wave radar and visual sensors, which is particularly suitable for solving the perception robustness problem in complex traffic scenarios (such as sudden changes in illumination, rainy and foggy weather, and occlusion). Background Technology

[0002] With the development of intelligent transportation systems, vehicle detection, as a key task of the perception layer, is directly related to road safety and traffic efficiency. Currently, target detection methods based on monocular vision are relatively mature; however, their accuracy and robustness still face significant bottlenecks in complex scenarios such as changing lighting, occlusion, rain, and snow. Millimeter-wave radar possesses all-weather operation capabilities and strong penetration, providing stable target distance and velocity information, but its spatial resolution is low, making it prone to false alarms and blurred target outlines. To address these issues, multimodal target detection methods based on the fusion of millimeter-wave radar and visual information have gradually emerged in recent years. By combining the complementary advantages of the two sensors, the accuracy and environmental adaptability of vehicle detection are improved. However, existing methods still have shortcomings in multi-scale feature fusion, modal alignment, and weight control, making it difficult to fully utilize the performance improvement potential brought by fusion. Therefore, there is an urgent need for a vehicle detection method that improves the fusion structure and attention mechanism to achieve more robust multimodal perception in complex scenarios. Summary of the Invention

[0003] The aim is to improve the accuracy, robustness, and consistency of multimodal perception of vehicle detection in complex traffic scenarios (especially under conditions of changing lighting, extreme weather, and occlusion).

[0004] To achieve the above objectives, the present invention adopts the following solution: In a first aspect, the present invention provides a data preprocessing method for vehicle detection based on the fusion of millimeter-wave radar and vision, comprising: The point cloud data of the millimeter-wave radar is acquired, the point cloud data is clustered, and the point cloud representing the vehicle target is extracted; the point cloud is then projected onto the image plane corresponding to the visual sensor to generate a single-channel binary mask image R0. Secondly, the present invention provides a data fusion method for vehicle detection based on the fusion of millimeter-wave radar and vision, including: A dual-input fusion network is constructed, with the input being a mask image R0 and an RGB image. Based on the R-PAN structure, the radar feature map (R3, R4, R5) and the visual feature map (P3, P4, P5) are spatially aligned, channel-stitched, and channel-unified at the corresponding scales to generate a fusion feature map (N3_R, N4_R, N5_R). Thirdly, the present invention provides a vehicle detection system based on the fusion of millimeter-wave radar and vision, comprising: The fused feature map is applied to an improved dual cross-attention mechanism (DCA), which includes: a dynamic weighted channel cross-attention (DW-CCA) module for adaptively adjusting the channel fusion weights according to ambient lighting; a deformable spatial cross-attention (DF-SCA) module for achieving adaptive alignment of spatial features; and a cross-modal consistency constraint (CMC) module for constraining the consistency of visual and radar modalities in semantics and attention distribution, in order to achieve deep enhancement of the fused features.

[0005] In practical implementation, Dynamic Weighted Channel Cross-Attention (DW-CCA) is used, which is based on the image ambient light intensity factor. This enables adaptive adjustment of the fusion weights of visual and radar modal features; Furthermore, the Dynamic Weighted Channel Cross-Attention (DW-CCA) module performs global average pooling on both visual and radar features along the channel dimension to obtain the channel description vector z, which is expressed by the formula:

[0006] Where H and W are the height and width of the input feature map, respectively. This indicates the spatial location of the c-th channel of the input feature map. The pixel value.

[0007] Use light regulation factor The illumination adjustment factor is an empirically set value, set to 0.001. It is the actual light intensity, expressed by the formula:

[0008] The adjusted channel vector z is then input into a two-layer fully connected network, and channel weight vectors for fused features are generated using ReLU and Sigmoid activation functions. This can be expressed by the formula:

[0009] in and For learnable parameters, Represents the ReLU activation function. This represents the Sigmoid function.

[0010] Feature fusion based on illumination perception is achieved through dynamic weighting. The weighting of millimeter-wave radar is increased under low-light conditions, while the weighting of vision is increased under normal illumination. This adaptive capability improves the model's performance under different lighting conditions, as expressed by the formula:

[0011] in To generate the channel weight vector of the fused features, Enhance channel features.

[0012] The Deformable Spatial Cross Attention (DF-SCA) module, based on deformable convolution and offset field prediction, dynamically samples key regions in the fused feature map to achieve adaptive alignment in the spatial dimension. In practical implementation: The Deformable Spatial Cross Attention Module (DF-SCA) uses convolution operations to generate the offset field. Spatial offset, based on deformable convolution operation, at each reference location Weighted sampling is performed at the location.

[0013] Further, the deformation feature map is obtained, which can be expressed by the formula: , in Indicates the location Output feature value at, For the input feature map, For the first Each convolutional kernel weight, No. One standard sampling position offset, No. The learnable offset of each sampling point, where N is the number of sampling points.

[0014] right By applying convolution and the sigmoid activation function, a spatial attention map is generated, expressed by the formula: , in This is a feature map generated through deformable convolution.

[0015] The spatial attention map A is multiplied pixel-by-pixel with the deformation feature map and then weighted and fused to obtain the final spatial enhancement feature map. The formula is expressed as: ,

[0016] in For spatial enhancement features of the final output, This is a diagram showing the deformation characteristics.

[0017] Step 4: Cross-modal consistency constraint (CMC), which calculates semantic similarity loss. KL divergence loss with attention distribution This guides visual and radar features to maintain consistency in semantics and spatial dimensions.

[0018] In specific implementation: The Cross-Modal Consistency Constraint Module (CMC) calculates the semantic vector after a 1×1 mapping between visual features F and radar features R. And minimize its Euclidean distance.

[0019] Furthermore, the semantic consistency loss is obtained, expressed by the formula: , in For visual branch feature maps, For radar branch feature maps, Two 1×1 convolution mapping functions are used to align dimensions. It is the consistency loss (Euclidean distance) between the two modalities in terms of semantic features.

[0020] Attention maps were extracted for the visual modality and the radar modality, respectively. and The KL divergence is calculated as the attention distribution loss, expressed by the following formula: , in For visual branch attention maps, For radar branch attention map, Kullback-Leibler divergence is used to measure the difference between two probability distributions. The loss is the distribution consistency loss between attention maps.

[0021] Finally, the overall consistent loss function is constructed, expressed by the formula:

[0022] in The total loss for cross-modal consistency constraints. This is the loss weighting factor. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the process of fusing millimeter-wave radar and vision.

[0024] Figure 2 This is a diagram showing the relationship between different coordinate systems.

[0025] Figure 3 This is a schematic diagram of the R-PAN algorithm of the present invention.

[0026] Figure 4 This is a schematic diagram of the improved YOLOv5 algorithm of the present invention.

[0027] Figure 5This is a schematic diagram of the DCA algorithm structure of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0029] A vehicle detection method based on millimeter-wave radar and vision fusion is proposed. This method successfully fuses millimeter-wave radar information. First, an RGB image frame and corresponding millimeter-wave radar point cloud data are input. The point cloud data is clustered using DBSCAN to extract vehicle target point clouds, and then projected to generate a single-channel binary mask image. Subsequently, a dual-input fusion network is constructed to extract multi-scale features from the mask image and the RGB image respectively. Multi-scale fusion is achieved through an R-PAN structure to obtain a fused feature map. Based on this, an improved dual-cross attention mechanism (DCA) is introduced, including a DW-CCA module that adaptively adjusts modal weights according to ambient lighting, a DF-SCA module for spatial alignment, and a CMC module that maintains consistency between semantics and attention distribution, to deeply enhance the fused features. The final output fused features are used in a vehicle detection model to achieve higher accuracy and robustness in multi-modal target detection in complex scenes. The flowchart is shown below. Figure 1 As shown. This effectively solves the problem of reduced detection performance of vision under extreme weather conditions, significantly improving the robustness of vehicle detection. The fusion method implementation steps are as follows: Step 1: Acquire sparse point cloud data from millimeter-wave radar, cluster the point cloud using the density clustering algorithm DBSCAN, and extract the target point cloud for each vehicle.

[0030] In practice, the point cloud data includes the spatial three-dimensional coordinates (x, y, z) and reflection intensity value (RCS) of each reflection point. To improve the quality of the point cloud, the original point cloud is first pre-screened, retaining only the points located within the effective detection area.

[0031] Furthermore, the core parameter of the DBSCAN clustering algorithm is set as: neighborhood search radius. =0.5 meters, minimum number of samples MinPts = 5. In the clustering results, each point cloud cluster corresponds to one vehicle target, and other isolated points are considered noise and discarded.

[0032] Step 2: Project each point cloud cluster from 3D space onto the image plane according to the preset projection calibration relationship to generate a single-channel binary mask image R0 with the same resolution as the visual image.

[0033] In practice, the projection process is based on the extrinsic parameter matrix of the millimeter-wave radar and the camera. Specifically, the millimeter-wave radar and the camera are calibrated to synchronize the millimeter-wave radar with vision in time and space.

[0034] In practice, the millimeter-wave radar sampling frequency is 80ms, and the camera sampling frequency is 40ms. Time synchronization is achieved by adjusting the camera sampling frequency to 80ms.

[0035] Furthermore, the spatial synchronization specifically involves: camera calibration, which is a process of establishing the relationship between the real scene and camera pixels based on an imaging model. A common method is to use a calibration board of consistent size and take images from different angles and positions, and then calibrate and calculate the parameters. Through MATLAB calibration, the final camera extrinsic parameters, rotation matrix 𝑹 and relative displacement vector 𝑻, are obtained.

[0036] Transform the world coordinate system to the camera coordinate system, and then transform the camera coordinate system coordinates to the image pixel coordinate system. The relationship between these three is as follows: Figure 2 As shown. World coordinate system. Transform into image pixel coordinate system The relation is , In the formula: The Z-axis coordinate in the camera coordinate system; These are the coordinates of the projection point in the image coordinate system onto the pixel coordinate system. The camera extrinsic rotation matrix is ​​a 3×3 orthogonal matrix with 3 degrees of freedom; It is a relative displacement vector; The focal length of the camera; These represent the unit physical quantities of the x and y axes of the pixel coordinate system for each pixel. The formula for transforming the radar coordinate system to the world coordinate system centered on the camera is: , Step 3: Construct a dual-input fusion network and achieve multi-scale cross-fusion of radar and visual features through an improved R-PAN structure; In a specific implementation, the obtained single-channel mask image R0 is input to the radar branch network, and radar feature maps R3, R4 and R5 of different scales are extracted sequentially through five layers of max pooling operation; at the same time, the RGB image is input to the visual branch network, and visual feature maps P3, P4 and P5 of the corresponding scale are extracted using the YOLOv5 backbone structure.

[0037] Furthermore, R3 and P3, R4 and P4, and R5 and P5 are aligned in the spatial dimension and then concatenated in the channel dimension. A 1×1 convolution is then used to unify the number of channels, resulting in fused feature maps N3_R, N4_R, and N5_R, as shown below. Figure 3 As shown in the diagram. The overall improved YOLOv5 network structure is as follows. Figure 4 As shown.

[0038] Step 4: Based on the fused feature map, an improved dual cross attention mechanism (DCA) is introduced to further enhance the expressive power of radar and visual modal fusion features in terms of channel dimension, spatial dimension, and semantic consistency. In its specific implementation, the improved DCA includes a Dynamic Weighted Channel Cross-Attention (DW-CCA) module for adaptively adjusting channel fusion weights based on ambient lighting; a Deformable Spatial Cross-Attention (DF-SCA) module for achieving adaptive alignment of spatial features; and a Cross-Modal Consistency Constraint (CMC) module for constraining the consistency of visual and radar modalities in semantics and attention distribution, such as... Figure 5 As shown; Step 5, Dynamic Weighted Channel Cross-Attention (DW-CCA) module, dynamically adjusts channel-level fusion weights in a data-driven manner, and improves the model's robustness to changes in lighting by introducing ambient light intensity as prior knowledge.

[0039] In its implementation, the DW-CCA module performs global average pooling of visual and radar features along the channel dimension to obtain the channel description vector z. Furthermore, this can be expressed as a formula:

[0040] Where H and W are the height and width of the input feature map, respectively. This indicates the spatial location of the c-th channel of the input feature map. The pixel value.

[0041] Use light regulation factor The illumination adjustment factor is an empirically set value, set to 0.001. This is the actual light intensity, obtained through OpenCV, and expressed by the formula:

[0042] The adjusted channel vector z is then input into a two-layer fully connected network, and channel weight vectors for fused features are generated using ReLU and Sigmoid activation functions. This can be expressed by the formula: , in and For learnable parameters, This represents the Sigmoid function. This represents the ReLU function.

[0043] Feature fusion based on illumination perception is achieved through dynamic weighting. The weighting of millimeter-wave radar is increased under low-light conditions, while the weighting of vision is increased under normal illumination. This adaptive capability improves the model's performance under different lighting conditions, as expressed by the formula: , in To generate the channel weight vector of the fused features, Enhance channel features.

[0044] Step 6: Deformable Spatial Cross Attention (DF-SCA) module, dynamically adjusts the receptive field of the attention mechanism to enhance the model's receptive field for complex scenes.

[0045] In practical implementation, this module is used to dynamically model spatial regions with fused features, enhance the perception of structural information such as vehicle edges and blurred contours, and improve spatial alignment accuracy.

[0046] Further improvements to the DF-SCA include: (1) Deformable convolution sampling: Predicting the migration field through deformable convolution. At each reference position Spatial sampling is performed at the location: , (2) Spatial attention generation: The deformed convolution result is input into the convolutional layer and activated by Sigmoid to obtain the spatial attention map. : , (3) Spatial Augmentation Feature Output: The attention map is combined with the original deformable feature map to output the final spatial augmentation feature: , Step 7: Cross-modal consistency constraint (CMC) module, which improves the robustness of fused features by explicitly aligning the high-level semantics and spatial distribution of visual and radar features.

[0047] In specific implementation: The Cross-Modal Consistency Constraint Module (CMC) calculates the semantic vector after a 1×1 mapping between visual features F and radar features R. And minimize its Euclidean distance.

[0048] Furthermore, the semantic consistency loss is obtained, expressed by the formula: ,

[0049] in For visual branch feature maps, For radar branch feature maps, Two 1×1 convolution mapping functions are used to align dimensions. It is the consistency loss (Euclidean distance) between the two modalities in terms of semantic features.

[0050] Attention maps were extracted for the visual modality and the radar modality, respectively. The KL divergence is calculated as the attention distribution loss, expressed by the following formula: ,

[0051] in For visual branch attention maps, For radar branch attention map, Kullback-Leibler divergence is used to measure the difference between two probability distributions. The loss is the distribution consistency loss between attention maps.

[0052] Finally, the overall consistent loss function is constructed, expressed by the formula: ,

[0053] in The total loss for cross-modal consistency constraints. This is the loss weighting factor.

Claims

1. A vehicle detection method based on millimeter-wave radar and vision fusion, characterized in that, Includes the following steps: S1. Acquire point cloud data from millimeter-wave radar, perform clustering processing on the point cloud data, extract point clouds representing vehicle targets, and project the point cloud onto the image plane corresponding to the visual sensor to generate a single-channel binary mask image R0. S2. Construct a dual-input fusion network, with the first input being the mask image R0 generated in step S2 and the second input being an RGB image; based on the R-PAN structure, perform spatial alignment, channel stitching, and channel unification operations on the radar feature map extracted from the mask image R0 and the visual feature map extracted from the RGB image at multiple corresponding scales to generate a cross-modal fusion feature map; S3. The cross-modal fusion feature map generated in step S2 is input into the improved dual cross-attention mechanism DCA for processing. The DCA mechanism includes: S3.1 Dynamic Weighted Channel Cross-Attention (DW-CCA) module: used to adaptively adjust the fusion weights of the radar features and the visual features in the channel dimension according to the ambient light intensity; S3.2 Deformable Spatial Cross Attention (DF-SCA) module: used to dynamically sample key regions in the fused feature map through deformable convolution, achieving adaptive feature alignment in spatial dimensions; S3.3 Cross-modal consistency constraint (CMC) module: used to calculate and constrain the consistency between the visual features and the radar features at the high-level semantic level and the spatial attention distribution level; S4. Based on the enhanced fusion features processed by the DCA mechanism in step S3, perform vehicle target detection and output the detection results.

2. The vehicle detection method according to claim 1, characterized in that, First, the two sensors are calibrated to ensure temporal and spatial synchronization. Second, the R-PAN structure described in step S2 is as follows: at multiple preset scales, the radar feature map extracted by the radar branch and the visual feature map extracted by the visual branch are spatially aligned, then stitched together in the channel dimension, and a convolution operation is used to unify the channel dimension to generate the cross-modal fusion feature map.

3. The vehicle inspection method according to claim 1, characterized in that, The deformable spatial cross-attention (DF-SCA) module is configured to: predict the spatial offset field, perform deformable convolution sampling on the input feature map, and generate a deformable feature map. A spatial attention map is generated based on the deformation feature map; The spatial attention map is combined with the deformation feature map to output a spatial enhancement feature map.

4. The vehicle detection method according to claim 1, characterized in that, The Cross-Modal Consistency Constraint (CMC) module is configured to calculate the distance between visual features and radar features in the mapped high-level semantic vectors as the semantic consistency loss. The distribution difference between the attention maps extracted by the computational vision branch and the radar branch is used as the attention distribution consistency loss; the total consistency loss is constructed based on the semantic consistency loss and the attention distribution consistency loss.

Images