Intelligent traffic target detection system and method based on 4D millimeter wave and laser radar point cloud fusion

Abstract

The application relates to the technical field of automatic driving and belongs to an intelligent traffic target detection system and method based on 4D millimeter wave and laser radar point cloud fusion; the system comprises a 4D millimeter wave radar motion feature enhancement module, a point cloud bidirectional fusion coding module, a sparse grid attention module and a multi-scale cross-attention fusion network module; the 4D millimeter wave radar motion feature enhancement module can enhance the dynamic information representation capability of point clouds; the point cloud bidirectional fusion coding module can effectively fuse laser and 4D millimeter wave radar point cloud voxel information, bridge the difference between heterogeneous data and realize the complementation of multi-modal information; the sparse grid attention module can capture global correlation information of a sparse BEV grid and improve the sensing efficiency; the multi-scale cross-attention fusion network fuses multi-scale BEV features based on a cross-attention unit, so that the effective interaction of information between fine-grained modes is realized.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to an intelligent traffic target detection system and method based on 4D millimeter wave and lidar point cloud fusion. Background Technology

[0002] In autonomous driving systems, environmental perception is a crucial prerequisite for intelligent decision-making, and its accuracy and robustness directly determine vehicle safety and decision reliability. LiDAR, with its high-precision three-dimensional ranging capabilities, can effectively acquire rich environmental geometric information, thus being widely used in the perception modules of autonomous driving systems. However, LiDAR is susceptible to environmental factors; in low-visibility scenarios such as rain, fog, snow, and sandstorms, its signal attenuation is significant, resulting in a marked decrease in detection range and accuracy. Furthermore, LiDAR is costly, generates massive amounts of data, and has high computational complexity, thus limiting its large-scale application in autonomous driving systems to some extent. In contrast, 4D millimeter-wave radar utilizes electromagnetic waves to perceive the environment, possessing excellent penetration and anti-interference capabilities. It can operate stably in adverse weather and low-light environments. It can not only effectively estimate the target's distance, azimuth, and elevation angles but also accurately perceive the target's Doppler information, thus providing crucial support for dynamic target detection and tracking. However, despite the significant advantages of millimeter-wave radar in scene adaptability, its scene perception accuracy is still far inferior to that of lidar. Therefore, it struggles to accurately depict the contours and texture details of complex targets, making it difficult to obtain detailed target perception results. In conclusion, due to inherent performance limitations, single-type sensors cannot meet the high-precision, all-scenario perception requirements of autonomous driving under diverse traffic conditions.

[0003] Studies have shown that even under normal weather conditions, multi-sensor fusion technology significantly outperforms single-sensor solutions in terms of target detection accuracy, system robustness, and environmental adaptability. The complementary advantages of different sensors are particularly pronounced when vehicles are in complex weather conditions such as rain, fog, snow, and sandstorms, or in extreme lighting environments. By fusing information from multiple sources, such as LiDAR and 4D millimeter-wave radar, the blind spots of single sensors can be effectively compensated for, thereby improving the stability and reliability of autonomous driving in complex scenarios.

[0004] Traditional multi-sensor fusion methods primarily rely on mathematical tools such as geometric modeling, probability statistics, and multidimensional filtering. The process typically includes: unifying multi-sensor observations to a single coordinate system through extrinsic parameter calibration and coordinate transformation; then using probabilistic models such as Bayesian inference, occupancy grids, and Markov random fields to characterize the uncertainties of multi-source measurements and complete environmental state estimation; finally, achieving temporal fusion and continuous target state inference based on tracking models such as Kalman filtering and particle filtering. While traditional fusion methods offer good interpretability and stability, they rely on strict model assumptions and lack adaptability to complex scenarios such as noise, occlusion, and high mobility, easily leading to decreased estimation and prediction performance or even filter divergence, thus making them unsuitable for handling complex and ever-changing real-world traffic environments.

[0005] With the development of data-driven point cloud feature extraction technology, deep learning-based multi-sensor fusion has become an important research direction in the field of autonomous driving perception. Based on the different fusion stages and data formats, existing methods are mainly divided into three categories: point cloud fusion, feature fusion, and decision fusion.

[0006] Point cloud fusion methods perform multimodal feature interaction at the point or voxel level, aiming to fully utilize the complementary information of lidar and 4D millimeter-wave radar. A typical approach introduces an attention mechanism after voxelizing the point cloud, performing explicit cross-modal association learning on the columnar voxel features of lidar and 4D millimeter-wave radar to enhance feature representation capabilities in downstream 3D target detection tasks. Further research refines the fusion to the voxel neighborhood level, using non-empty lidar voxels as anchors and adaptively enhancing lidar features with neighborhood millimeter-wave radar voxels, thereby improving the detection performance of long-range dynamic targets. However, 4D millimeter-wave radar point clouds generally suffer from clutter noise caused by multipath effects and environmental reflections. During point or voxel-level fusion, while radar information enhances perception capabilities, it may propagate noise to deeper feature spaces, affecting the robustness of feature representation. To address this issue, some studies introduce a foreground-aware denoising mechanism before fusion to suppress background clutter in radar echoes and achieve bidirectional feature fusion at the columnar voxel level to improve clutter robustness under complex weather conditions. However, such methods typically only use attributes such as velocity as additional dimensions in voxel encoding, failing to fully leverage the advantages of 4D millimeter-wave radar in perceiving moving targets in complex dynamic scenes.

[0007] Feature fusion methods focus on fusing deep features from bird's-eye view (BEV) images. Early works used channel stitching to fuse BEV features from lidar and millimeter-wave radar. This method is simple and effective, but it does not consider the differences in key information between different modes, potentially introducing target-irrelevant redundant information. To improve this, some studies enhance the cross-modal features of key regions and their neighborhoods by predicting potential target locations, thus focusing the fusion on target-related regions. Other works introduce adaptive weighting modules based on spatial feature statistics to dynamically adjust the contributions of the two modes, mitigating the imbalance in modal contributions caused by the sparse point clouds and high noise of millimeter-wave radar. Further research extends BEV feature fusion to degraded scenarios such as severe weather, proposing multi-branch parallel backbone networks to extract intra-modal and inter-modal features separately, and dynamically adjusting the fusion weights of different modes through gating mechanisms. However, these methods largely rely on feature alignment or weight allocation strategies, making it difficult to fully integrate the indicative information unique to millimeter-wave radar, such as velocity and radar cross section (RCS), as well as the fine geometry of lidar. Subsequent research has constructed a bidirectional mutual sensing fusion mechanism at the BEV feature level, enabling radar indication information and laser geometric features to mutually guide and enhance each other, thereby improving cross-modal complementary expression capabilities. However, existing methods typically assume that the BEV features of millimeter-wave radar can be effectively extracted using convolutional networks consistent with those of lidar, ignoring the extreme sparsity and uneven distribution of effective information in the BEV space, thus limiting the effective transmission and expression of radar-specific information.

[0008] Decision fusion methods typically fuse the detection results of two modalities at the decision level based on their confidence levels. For example, they combine the platform motion state with the detection results of cross-modal uncertainty-enhanced LiDAR and fuse the output detection boxes of the two modalities to correct the final decision. However, the performance of decision-level fusion largely depends on the independent detection capabilities of each branch. Due to the lack of interaction between the original data and deep features, its overall performance is usually inferior to the first two types of methods.

[0009] In summary, existing fusion methods still have certain limitations in terms of information interaction depth, feature representation efficiency, and modal characteristic adaptation. There is an urgent need to develop efficient fusion strategies that can more effectively combine the geometric information of LiDAR and the indicative information of 4D millimeter-wave radar to improve the robustness and accuracy of autonomous driving perception systems in complex environments. Summary of the Invention

[0010] To address the aforementioned technical problems, this invention provides an intelligent traffic target detection system and method based on 4D millimeter wave and lidar point cloud fusion.

[0011] To achieve the above objectives, the present invention provides the following technical solution: On one hand, the present invention provides an intelligent traffic target detection system based on 4D millimeter-wave and lidar point cloud fusion, comprising: The 4D millimeter-wave radar motion feature enhancement module is used to extend the radial velocity obtained by point cloud compensation and construct a point cloud motion state prediction network that focuses on velocity features to enhance the distinction between dynamic and static point clouds. The point cloud bidirectional fusion encoding module is used to fuse point cloud voxel information from laser and 4D millimeter-wave radar. It performs bidirectional fusion encoding on the two point clouds within the columnar voxel, bridging the differences between heterogeneous data and achieving multimodal information complementarity, and generating BEV pseudo-image features. The sparse mesh attention module uses sparse masks to perform attention calculations on non-empty meshes to obtain new pseudo-image features, which are used to capture global correlation information of sparse 4D millimeter-wave radar BEV meshes. The multi-scale cross-attention fusion network module fuses multi-scale BEV features based on cross-attention units to achieve information interaction between fine-grained modalities and obtain a fused feature map. The detection head module receives the fused feature map and performs final detection.

[0012] Preferably, the processing sub-steps in the 4D millimeter-wave radar motion feature enhancement module are as follows: S11. Decompose the target's absolute radial velocity to obtain rich original point features with velocity components; S12. Perform multiple data processing on the obtained velocity components to generate channel-by-channel attention weights; S13. Multiply the obtained channel-by-channel attention weights by the original point features to apply differentiated weights to different speed feature channels; S14. The weighted features are added to the original features through residual connections to obtain enhanced features; S15. Perform feature extraction and point-level segmentation on the obtained enhanced features to obtain the point cloud motion state prediction result. Embed this motion state as a new feature into the point cloud representation to obtain extended features.

[0013] Preferably, the specific steps of the multiple data processing in step S12 are as follows: First, the velocity features are compressed through a one-dimensional convolutional layer. Then, non-linear representations are introduced through batch normalization and ReLU activation function. Subsequently, the features are mapped to the original channel dimension through a one-dimensional convolutional layer. Finally, the output is normalized based on the Sigmoid function to generate channel-wise attention weights.

[0014] Preferably, the processing sub-steps in the point cloud bidirectional fusion encoding module are as follows: S21. Based on the Pointpillars model, the LiDAR point cloud is divided into columnar voxels and encoded. S22. Using the method in step S21, obtain the 4D millimeter-wave radar point cloud code; S23. Based on the two encodings obtained in steps S21 and S22, embed their respective key features into the voxels to complete point cloud fusion; S24. Aggregate point cloud features within non-empty voxels through linear layers and max pooling layers to obtain columnar features; S25. Based on the columnar features obtained in step S24 and according to the voxel features in the planar grid position index, map them to the corresponding two-dimensional grid cells to form the corresponding voxel-level high-dimensional feature tensor. S26. Fill the empty voxel grid with zero values ​​and obtain the corresponding tensor; S27. Combine the tensors obtained in steps S25 and S26 to generate multidimensional BEV pseudo-image features.

[0015] Preferably, the processing sub-steps in the sparse mesh attention module are as follows: S31. Based on the BEV pseudo-image features obtained in the point cloud bidirectional fusion coding module, generate a sparse mask matrix. S32. Flatten the mask and obtain N non-empty grid features according to the index, construct a compact feature sequence, and use this as the valid grid token; S33. Input the compact feature sequence obtained in step S32 into the self-attention layer, and use a multilayer perceptron to map the channel of dimension C to the attention space of hidden dimension D, and output the compact feature sequence of dimension D. S34. Perform LayerNorm operation on the compact feature sequence in dimension D obtained in step S33. The results are linearly mapped through three independent multilayer perceptrons to obtain query vector, key vector and value vector. S35. Based on the vector obtained in step S34, calculate the sparse attention; S36. The sparse attention obtained in step S35 and the compact feature sequence obtained in step S33 are fed to the feedforward network and the residuals are added again to obtain the enhanced features. S37. The channel features of dimension D are projected onto the original space of dimension C through a multilayer perceptron to obtain the updated compact feature sequence. S38. Using the grid index recorded in step S32, restore the updated compact feature sequence to a pseudo-image grid of size H×W, thereby obtaining a new pseudo-image feature map.

[0016] Preferably, the multi-scale cross-attention fusion network module includes: The multi-scale feature extraction module uses a three-layer convolutional network to downsample the features of the two-modal pseudo-images to different scales to obtain multi-scale features. The cross-attention feature fusion module performs feature interaction on the same scale features of two modalities among the obtained multi-scale features. Each scale uses a cross-attention unit to process and calculate to generate highly separable feature vectors at specific locations. The two-branch cross-attention unit uses independent parameter configuration. The feature upsampling fusion module upsamples the obtained multi-scale feature maps to the same spatial resolution. At the same time, it aligns features of different scales to the same scale to facilitate subsequent fusion. The multi-scale feature maps are stitched together along the channel dimension to finally form the fused feature map that serves as the input to the detection head.

[0017] Preferably, the cross-attention unit processing calculation process is as follows: S421. Perform dimension alignment and dimensionality reduction operations on LiDAR features and 4D millimeter-wave radar features through two convolutional layers with the same kernel size, while preserving key semantic information to generate query vectors and key features. S422. The obtained dimensionality reduction features are used to construct a spatial correlation matrix through matrix dot product operation and normalized by the softmax function to characterize the similarity measurement between different modal features in each spatial location. Simultaneously, the context information of the LiDAR features is aggregated based on the convolutional layer. S423. Obtain lidar enhancement features about the feature map location by weighted aggregation of 4D millimeter-wave radar feature context information; S424. The 4D millimeter-wave radar enhancement feature is generated by processing steps S421-S423. S424. By splicing enhanced features of different scales according to the channel dimension and adjusting the dimension through the convolutional layer of the convolutional kernel, scale fusion features are obtained.

[0018] On the other hand, the present invention also provides a method for detecting intelligent traffic targets based on deep sparse representation applicable to the above-mentioned system, comprising the following steps: S1. Extract target motion features based on dynamic feature enhancement module, expand the radial velocity obtained by point cloud compensation, and construct a point cloud motion state prediction network focusing on velocity features. S2. Embed the point cloud features of 4D millimeter wave and lidar to bridge the differences in heterogeneous modal data. At the same time, extract voxel-level features using column voxels as units and project them onto a 2D plane to obtain BEV pseudo-images. S3. Use sparse masking to perform attention calculation only on non-empty grids, extract 4D millimeter-wave radar pseudo-image features, and obtain new pseudo-image features. The sparsity construction based on BEV features can effectively extract sparse mesh attention modules for this feature. S4. Multi-scale multimodal features are fused through a multi-scale cross-attention fusion network, thereby integrating complementary information from different granular levels of multimodal features to form a fused feature map that can be used as input to the detection head. S5. Input the obtained fused feature map into the detection head to obtain high-confidence detection of 3D targets.

[0019] Compared with existing technologies, the present invention provides an intelligent traffic target detection system and method based on 4D millimeter wave and lidar point cloud fusion, which has the following beneficial effects: 1. The system provided by this invention includes a 4D millimeter-wave radar motion feature enhancement module that enhances the dynamic information representation capability of point clouds; a point cloud bidirectional fusion coding module that effectively fuses laser and 4D millimeter-wave radar point cloud voxel information, bridging the differences between heterogeneous data and achieving multimodal information complementarity; a sparse grid attention module that captures global correlation information of sparse BEV grids while improving perception efficiency; and a multi-scale cross-attention fusion network that fuses multi-scale BEV features based on cross-attention units, thereby achieving effective interaction of information between fine-grained modalities.

[0020] 2. Experiments based on View-of-Delft and its fog simulation dataset show that, compared with lidar-based methods and existing mainstream fusion methods, the detection method provided by this invention significantly improves the mAP3D and mAPBEV values ​​obtained under normal weather conditions and under foggy conditions. The method has high detection accuracy and robustness.

[0021] The features and advantages of the present invention will be described in detail through embodiments and in conjunction with the accompanying drawings. Attached Figure Description

[0022] Figure 1 This is a general framework diagram of the intelligent traffic target detection system based on 4D millimeter wave and lidar point cloud fusion of the present invention; Figure 2 This is a diagram of the radar dynamic feature enhancement module architecture of the present invention; Figure 3 This is a velocity decomposition diagram of the present invention; Figure 4 This is a diagram of the sparse mesh attention module architecture of the present invention; Figure 5 This is a diagram of the multi-scale cross-attention fusion network module architecture of the present invention; Figure 6 This is a network diagram of the attention unit of the present invention; Figure 7 The image shows the detection results obtained from the VoD dataset based on the detection method and comparison model of this invention. Figure 8 The images show the detection results obtained from the VoD fog simulation dataset for the detection method and comparison method of this invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0026] Example 1: like Figure 1 As shown, this embodiment provides a road target detection system based on 4D millimeter-wave and lidar point cloud fusion, which mainly consists of the following modules: (1) 4D millimeter-wave radar motion feature enhancement module, which can enhance the dynamic information representation capability of point cloud.

[0027] (2) Point cloud bidirectional fusion coding module, which can effectively fuse laser and 4D millimeter-wave radar point cloud voxel information, bridge the differences between heterogeneous data and realize multimodal information complementarity; (3) Sparse mesh attention module, which can capture global correlation information of sparse BEV mesh and improve perception efficiency; (4) Multi-scale cross-attention fusion network, which is based on the constructed FAC module to fuse multi-scale BEV features, thereby realizing the effective interaction of information between fine-grained modalities and obtaining fused feature maps.

[0028] (5) Detection head module, which receives the fused feature map and performs final detection.

[0029] The following section details each module of the proposed method: 1. Millimeter-wave radar dynamic feature enhancement module The radial velocity features of 4D millimeter-wave radar contain rich target motion information. In complex scenarios such as heavy fog, motion state can provide valuable supplementary information for target detection. However, existing target detection algorithms typically treat the one-dimensional radial velocity as auxiliary information when processing the velocity features of 4D millimeter-wave radar. This radial velocity only retains the velocity projection of the target along the radar's line-of-sight, making it difficult to fully characterize the target's true motion state. Furthermore, the target motion state distribution shown in Table 1 indicates that most weak targets within the detection area are in motion. Therefore, their velocity information can be fully utilized to enhance target discrimination and mitigate the impact of occlusion and background noise on detection. Based on this observation, motion state can be incorporated as an additional dynamic semantic constraint into the feature representation, thereby enhancing the dynamic contours of weak targets and compensating for the lack of dynamic information in long-range, weakly textured areas, or low-visibility environments using lidar. Therefore, this section constructs... Figure 2 The millimeter-wave radar dynamic feature enhancement module shown includes three sub-modules: velocity decomposition, velocity attention calculation, and feature extraction.

[0030] Table 1 Distribution of object motion states in VoD dataset

[0031] Specifically, assuming the input is a point cloud Where N is the number of radar point clouds, and the information of each point includes three-dimensional coordinates. Relative and absolute radial velocities , The absolute radial velocity, RCS, and timestamp t are used. The absolute radial velocity can be obtained through platform self-motion compensation, which more effectively reflects the target's motion state. Therefore, in a Cartesian coordinate system with the millimeter-wave radar as the origin, this section will use the absolute radial velocity... Perform as Figure 3 The orthogonal decomposition shown obtains the following velocity components, thereby enriching the features of the original point: (1) (2) in, The absolute radial velocity component in the x-direction. The absolute radial velocity component in the y-direction. This is the target's azimuth angle relative to the radar.

[0032] Based on the obtained velocity components, a velocity attention module is constructed to adaptively enhance point clouds with significant motion characteristics. Specifically, the input velocity-encoded features are first compressed through a one-dimensional convolutional layer to reduce computational complexity. Then, nonlinear representations are introduced through batch normalization and the ReLU activation function. Finally, the features are mapped back to the original channel dimension through a one-dimensional convolutional layer. Finally, the output is normalized to the (0, 1) interval based on the Sigmoid function to generate channel-wise attention weights.

[0033] The obtained attention weights are multiplied element-wise with the original input features to achieve differentiated weighting of different speed feature channels. Then, the weighted features are added to the original features through residual connections to obtain enhanced features.

[0034] The obtained features are then input into a feature extraction module consisting of a lightweight PointNet network and a point-level segmentation head, and its output is the point cloud motion state prediction result. Where 0 and 1 represent static and dynamic states, respectively, this state can be used as a new feature to embed into the point cloud representation. To obtain extended features .

[0035] During model training, point cloud motion labels can be generated based on the target motion state. To effectively characterize the accuracy of motion state prediction, the following cross-entropy loss function is used to define the motion state loss, which measures the difference between the predicted value and the true label: (3) in, For the first Real-time motion tags at each point This represents the predicted motion probability at that point, where N is the number of samples.

[0036] 2. Point cloud bidirectional fusion encoding module Existing laser and 4D millimeter-wave radar fusion methods typically encode point clouds within their respective modal independent voxel spaces. This independent encoding strategy may lead to misalignment of feature semantics between the two, making it difficult to fully utilize complementary information. Especially under adverse weather conditions, the sparsity of laser point clouds and the increase in radar point cloud noise exacerbate the feature distribution shift between modalities, severely affecting the perception effectiveness of the fusion sensing module. To address this issue, this section proposes a bidirectional fusion encoding method for point clouds within columnar voxels.

[0037] Compared to traditional voxel feature fusion, bidirectional fusion can inject precise geometric information obtained from LiDAR into the millimeter-wave radar point cloud within a shared voxel space to compensate for its insufficient spatial resolution. Simultaneously, it uses radial velocity and robustness features obtained from 4D millimeter-wave radar to inversely enhance the LiDAR point cloud, improving its perception capability in degraded scenes. This cross-modal feature complementarity mechanism can achieve modal alignment in the initial feature extraction stage, effectively suppressing fusion noise and improving multimodal fusion results. The proposed method divides the LiDAR point cloud into columnar voxels and encoding based on the Pointpillars model, with the voxel set represented as... , For each voxel, the point cloud is set. Given the total number of voxels, the point cloud representation obtained by encoding within each voxel can be represented as follows: (4) in, In three-dimensional space coordinates, The average distance from the point cloud to all points within the voxel is given. This represents the horizontal position offset of the point cloud from the voxel center. This represents the reflection intensity. This relative position encoding effectively enhances the model's ability to learn about local geometry.

[0038] Similarly, the encoded point cloud of 4D millimeter-wave radar can be represented as: (5) in, , and Similar to equation (4), This is the collection of velocity features from the aforementioned millimeter-wave radar dynamic feature enhancement module. For RCS, This represents the predicted motion state of the point cloud. Based on the obtained encoded features, key features can be embedded into the common elements of both points, thus the fused point cloud can be represented as: (6) (7) in, , , and This represents the average value of all point cloud features within a voxel. At this point, bidirectional fusion encoding of multimodal point clouds can be achieved. Then, point cloud features within non-empty voxels are aggregated through linear layers and max-pooling layers to obtain the following columnar features: (8) (9) Based on the intra-column features obtained above, and according to the voxel features' position index on the planar grid, they are mapped to the corresponding two-dimensional grid cells, thereby constructing the corresponding voxel-level high-dimensional feature tensor. , ,in, This represents the number of Pillar feature channels. , These represent the number of non-empty voxels for laser and 4D millimeter-wave radar, respectively.

[0039] The empty voxels are filled with zero values ​​in the grid, resulting in tensors. , .

[0040] Based on the above, combined with , and , Based on the generation dimension BEV pseudo-image features , Where C is the number of channels, H is the height dimension, and W is the width dimension.

[0041] 3. Sparse mesh self-attention network The invention is constructed as follows Figure 4 The sparse grid self-attention module shown efficiently extracts features from 4D millimeter-wave radar pseudo-images. The constructed sparse self-attention module uses a sparse mask to perform attention calculations only on non-empty grids.

[0042] Specifically, for the obtained 4D millimeter-wave radar pseudo-image features First, generate a sparse mask matrix. , where if and only if the grid position When non-zero features exist Then flatten the mask and obtain the index. A number of non-empty grid features are used to construct a compact feature sequence. This is then used as a valid grid token. Since the token itself carries explicit location information, the model can learn spatial relationships based on a fixed grid order without additional location encoding. The resulting grid feature sequence is then... The input is fed into a self-attention layer similar to the Transformer architecture, which uses a Multilayer Perceptron (MLP) to map the C-dimensional channels to a D-dimensional attention space to produce the output. Regarding the results The LayerNorm operation is performed, and the results are linearly mapped through three independent MLPs to obtain the following query vectors. Key vector Sum value vector : (10) in, For learnable weight matrix, Hiding dimensions for attention. Based on this, sparse attention can be computed as follows: (11) in, This represents the scaling factor, and d represents the key vector. Dimensions.

[0043] Next, residual linkage and feedforward networks are used to enhance nonlinearity, that is: the obtained as well as The data is fed into a feed-forward neural network (FFN) and the residuals are summed again to obtain enhanced features. Subsequently, the D-dimensional channel features are projected onto the original C-dimensional space using an MLP to obtain an updated compact feature sequence. Finally, using the previously recorded grid index, Restore to size In the pseudo-image grid, i.e.: mask The position is filled with updated features. The empty grid is kept at zero, thus obtaining a new pseudo-image feature map. This feature map, while preserving the original spatial layout, aggregates global information between all non-empty grids through a self-attention mechanism, enabling subsequent modules to discriminate targets at a higher level. It's important to note that the attention stage only... The computation is performed on a non-empty grid, and the time and space complexity of self-attention is reduced by the standard Transformer. On each token steep drop to Furthermore, it concentrates gradient features in the effective region, thereby improving the detection effectiveness of 4D millimeter-wave radar.

[0044] 4. Multi-scale cross-attention fusion network As mentioned above, voxel-level fusion can effectively compensate for data differences between modalities and achieve primary feature complementarity, while mid-term BEV fusion can further map the high-level features of the two sensor data extracted by the dual-branch network to a unified bird's-eye view semantic space, achieving dual optimization of low-level feature alignment and high-level semantic complementarity, thereby maintaining spatial consistency while deeply associating high-level semantics.

[0045] Based on this, in order to fully leverage the respective advantages of laser and 4D millimeter-wave radar, the present invention is designed as follows: Figure 5The multi-scale cross-attention fusion network shown is used to fuse BEV features from two branches. It mainly consists of the following three parts: multi-scale feature extraction, FAC, and feature upsampling fusion module.

[0046] The multi-scale feature extraction module uses a three-layer convolutional network to extract the features of the input dual-branch pseudo-image. , Downsampling to features at different scales , , , Different modalities can be represented, allowing the model to simultaneously retain shallow local details and deep global semantic information, thereby improving the ability to represent targets at different scales and the robustness of detection.

[0047] Then, a cross-attention feature fusion module was designed to perform feature interaction on the obtained features of the same scale in the two modalities. Each scale uses two such features. Figure 6 The FAC unit is shown. Specifically, taking the lidar branch as an example, after processing by the cross-attention unit, the resulting enhanced lidar BEV feature can be expressed as: (12) in, This refers to the cross-attention unit of the lidar branch.

[0048] The two-branch cross-attention unit uses independent parameter configurations to avoid modal feature confusion caused by parameter sharing. The calculation process of the cross-attention unit is as follows: First, it uses two convolutional kernels with sizes of... The convolutional layers respectively target the features of the lidar. and 4D millimeter-wave radar characteristics Dimension alignment and dimensionality reduction are performed to reduce the computational complexity of subsequent attention operations, while preserving key semantic information to generate query vectors. Bond features The resulting dimensionality-reduced features are used to construct the following spatial correlation matrix through matrix dot product operations. and through Function normalization is used to characterize the similarity measure of different modal features across spatial locations: (13) Among them, the correlation matrix elements in Characterizing the location of lidar features With 4D millimeter-wave radar feature location The semantic association strength between them. Meanwhile, based on the convolution kernel size... The convolutional layer aggregates contextual information from LiDAR features, namely: (14) Finally, by weighted aggregation of 4D millimeter-wave radar feature context information, the following information about the feature map location is obtained. LiDAR enhancement features: (15) in, Location in millimeter-wave radar feature map The characteristic elements, This process indexes all locations in the millimeter-wave radar feature map. It filters complementary information useful for the current location from the millimeter-wave radar modes, then fuses it with the core information of the same mode to ultimately generate a highly separable feature vector specific to the location, thereby improving the target detection capability of the lidar. Similar to lidar, the 4D millimeter-wave radar BEV feature generated by another branch can be represented as: (16) Subsequently, enhanced features of different scales are concatenated according to the channel dimension, and the dimensions are adjusted by a convolutional layer with a kernel size of 1×1, thereby obtaining the following three scale fused features: (17) This fusion feature effectively integrates complementary information from two modalities at different granularity levels, thereby significantly enhancing the model's sensitivity to and ability to capture targets at different scales.

[0049] Finally, the upsampling and feature fusion module upsamples the acquired multi-scale feature maps to the same spatial resolution to enhance the spatial representation of low-resolution features, thereby improving the detection performance of small targets. Simultaneously, features at different scales are aligned to the same scale to facilitate subsequent fusion. Based on this, multi-scale feature maps are stitched along the channel dimension to effectively fuse information from different levels and receptive fields, ultimately forming a fused feature map that can be used as input to the detection head.

[0050] loss function This model is based on point-level motion attribute modeling of millimeter-wave radar and the collaborative improvement of system detection performance through two-modal fusion. Therefore, its loss function can be represented by a weighted sum of motion state and target detection losses. Among them, the motion state loss is shown in Equation (3), and the target detection loss function adopts the three-dimensional target detection loss design widely used in the existing technology, which consists of two parts: target classification loss and bounding box localization loss. The target detection loss is expressed as follows: (18) in, This represents the number of positive sample detection boxes. Classification loss for the target, To locate the regression loss for the target, and Here are the weighting coefficients. The classification loss can be expressed as: (19) in, Predict probabilities for each category. , These are the balance and focus parameters, respectively. The 3D ground truth box and prediction box parameters are... ,in The center position of the frame. To predict the length, width, and height of the bounding box, Let be the orientation angle. Based on this, the following localization regression loss can be constructed: (20) in, For smoothing loss function and This represents the truth box and the predicted regression results. The regression deviation between the predicted and truth boxes is shown in the following formula: (twenty one) (twenty two) (twenty three) Based on the above-mentioned target detection loss and loss of motion The total loss of the model can be constructed as follows: (twenty four) in, and These are the weighting coefficients.

[0051] Experimental Results and Analysis Based on the VOD dataset and its fog simulation dataset, this section verifies the effectiveness of the proposed method by comparing it with mainstream detection methods based on unimodal and multimodal fusion.

[0052] 1. Experimental setup The hardware and software environment for this experiment is shown in Table 2.

[0053] Table 2. Hardware and software environment and parameter configuration required for the experiment

[0054] 2. Comparative Experiment The detection results are evaluated using the Average Precision (AP) proposed by KITTI, with the Intersection over Union (IoU) thresholds for Car, Cyclist, and Pedestrian targets set at 0.5, 0.25, and 0.25, respectively. Based on VoD and its fog simulation dataset, this section evaluates the effectiveness of the proposed method by comparing it with the following six methods: LiDAR-based methods Pointpillars and PointRCNN; 4D millimeter-wave radar-based methods Pointpillars and PointRCNN; and fusion-based methods InterFusion and L4DR. It should be noted that the VoD fog simulation dataset only performs fog simulation processing on LiDAR data and does not introduce corresponding degradation processing for millimeter-wave radar data. Therefore, in the fog experiment setting, only the fusion method and the single LiDAR method are compared and analyzed to ensure fairness of different methods under consistent perception conditions and the interpretability of the comparison results.

[0055] 3. Comparative Experiment Detection results obtained from different models, such as Figure 7As shown in the figure, the first row and first column represent the scene image, the first row and second column represent the ground truth bounding boxes, and the rest represent the detection results of different methods. Black dots represent LiDAR point clouds, green dots represent 4D millimeter-wave radar point clouds, red boxes represent ground truth bounding boxes, and blue boxes represent detection bounding boxes. This figure shows that the single-modal methods PointPillars and PointRCNN based on 4D millimeter-wave radar exhibit significant missed detections in sparse point cloud regions, especially at target edges and in distant scenes. Detection results for small targets such as pedestrians and occluded vehicles are incomplete. In contrast, the single-modal methods PointPillars and PointRCNN based on LiDAR can detect near- and medium-range targets relatively accurately, with good spatial overlap between their predicted and ground truth bounding boxes. However, in distant dynamic scenes, they have a high rate of missed detections for small targets. Furthermore, the multi-modal fusion methods InterFusion and L4DR can detect most targets in scenes with sparse point clouds or occlusion. However, in complex scenes where static and dynamic targets are adjacent, there are still instances where static bicycles are misidentified as cyclists. In summary, the single-modal methods PointPillars and PointRCNN based on 4D millimeter-wave radar are highly sensitive to point cloud density and have limited ability to represent features in sparse regions. Meanwhile, the single-modal methods PointPillars and PointRCNN based on lidar lack the ability to perceive target motion information, leading to a significant degradation in their performance for detecting small, dynamic targets at long range. Multimodal fusion methods InterFusion and L4DR can improve detection completeness to some extent, but they fail to fully utilize the velocity information from 4D millimeter-wave radar, and their feature discrimination ability is relatively limited under sparse point cloud conditions, making them prone to false detections in complex scenes with moving and static targets in close proximity. In contrast, the proposed method shows almost no obvious missed or false detections, demonstrating high detection robustness. This can be attributed to the fact that the proposed method fully extracts target motion information through a dynamic feature enhancement module, effectively distinguishing between moving and static targets in the BEV space, thereby significantly reducing the risk of false detections due to semantic similarity or spatial proximity. Furthermore, the designed sparse grid attention network adaptively weights key spatial regions, effectively suppressing the noise response introduced by the sparsity of 4D millimeter-wave radar point clouds. This allows the model to focus on high-confidence regions relevant to target discrimination during the feature aggregation stage. Based on this, the model maintains the high-precision geometric perception advantage of LiDAR while incorporating dynamic supplementary information from millimeter-wave radar, thereby achieving robust and accurate target localization and category determination.

[0056] To further verify the robustness of the proposed method under complex weather conditions, target detection experiments were conducted on the VoD fog simulation dataset (fog concentration level 3) for each comparative method. The results are as follows: Figure 8As shown in the figure, the first row and second and third columns represent the point clouds before and after the lidar fog simulation, respectively. This figure reveals that under foggy conditions, the detection performance of lidar-based single-modal detection methods PointPillars and PointRCNN significantly decreases, only detecting a small number of near-range targets, while mid-to-long-range targets generally suffer from significant missed detections. In the fog simulation scenario, the multi-modal fusion methods InterFusion and L4DR perform relatively well overall, effectively detecting targets in some mid-to-long-range areas, but their performance is poor for moving targets in sparse lidar areas. As discussed above, lidar is prone to scattering and attenuation under foggy conditions, leading to a shortened effective sensing distance and a significant reduction in point cloud density. This weakens the ability of lidar-based single-modal sensing methods to model the target geometry during feature extraction, ultimately resulting in a significant degradation in mid-to-long-range detection performance. Furthermore, under conditions of severe degradation in lidar point clouds, multimodal fusion methods such as InterFusion and L4DR can fuse 4D millimeter-wave radar information to supplement discrimination information. However, they pay less attention to the velocity information extracted by the 4D millimeter-wave radar, and their ability to distinguish fine-grained targets is limited under sparse point cloud conditions, making it difficult to effectively distinguish adjacent moving and stationary targets. In contrast, the proposed method has good detection performance in dense fog scenarios, which can be explained as follows: The proposed method fully mines the motion information obtained by the 4D millimeter-wave radar through a dynamic feature enhancement module, and combines a sparse grid attention network to alleviate the deficiency of BEV features caused by the sparseness of millimeter-wave radar point clouds. Based on this, a multi-scale cross-attention fusion network is introduced to enhance the complementarity and correlation of cross-modal features at different scales, thereby improving the effectiveness and robustness of target detection under complex weather conditions.

[0057] Those skilled in the art will understand that the modules or steps described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, which can then be stored in a storage device for execution by a computer device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. This disclosure is not limited to any particular combination of hardware and software.

[0058] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0059] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent traffic target detection system based on 4D millimeter-wave and lidar point cloud fusion, characterized in that: include: The 4D millimeter-wave radar motion feature enhancement module is used to extend the radial velocity obtained from point cloud compensation and construct a point cloud motion state prediction network focusing on velocity features to enhance the distinction between dynamic and static point clouds. The processing sub-steps in the 4D millimeter-wave radar motion feature enhancement module are as follows: S11. Orthogonally decompose the target's absolute radial velocity to obtain rich original point features with velocity components; S12. Perform multiple data processing on the obtained velocity components to generate channel-by-channel attention weights; S13. Multiply the obtained channel-by-channel attention weights by the original point features to apply differentiated weights to different speed feature channels; S14. The weighted features are added to the original features through residual connections to obtain enhanced features; S15. Extract features and segment points into the obtained enhanced features to obtain the point cloud motion state prediction results. Embed this motion state as a new feature into the point cloud representation to obtain extended features. The point cloud bidirectional fusion encoding module is used to fuse point cloud voxel information from laser and 4D millimeter-wave radar. It performs bidirectional fusion encoding on the two point clouds within the columnar voxel, bridging the differences between heterogeneous data and achieving multimodal information complementarity, and generating BEV pseudo-image features. The sparse mesh attention module uses sparse masks to perform attention calculations on non-empty meshes to obtain new pseudo-image features, which are used to capture global correlation information of sparse 4D millimeter-wave radar BEV meshes. The multi-scale cross-attention fusion network module fuses multi-scale BEV features based on cross-attention units to achieve information interaction between fine-grained modalities and obtain a fused feature map. The detection head module receives the fused feature map and performs final detection.

2. The intelligent traffic target detection system based on 4D millimeter-wave and lidar point cloud fusion as described in claim 1, characterized in that: In step S12, the specific steps of multiple data processing are as follows: First, the velocity features are compressed through a one-dimensional convolutional layer. Then, non-linear representations are introduced through batch normalization and ReLU activation function. Subsequently, the features are mapped to the original channel dimension through a one-dimensional convolutional layer. Finally, the output is normalized based on the Sigmoid function to generate channel-wise attention weights.

3. The intelligent traffic target detection system based on 4D millimeter-wave and lidar point cloud fusion as described in claim 1, characterized in that: The processing sub-steps in the point cloud bidirectional fusion encoding module are as follows: S21. Based on the Pointpillars model, the LiDAR point cloud is divided into columnar voxels and encoded. S22. Using the method in step S21, obtain the 4D millimeter-wave radar point cloud code; S23. Based on the two encodings obtained in steps S21 and S22, embed their respective key features into the voxels to complete point cloud fusion; S24. Aggregate point cloud features within non-empty voxels through linear layers and max pooling layers to obtain columnar features; S25. Based on the columnar features obtained in step S24 and according to the voxel features in the planar grid position index, map them to the corresponding two-dimensional grid cells to form the corresponding voxel-level high-dimensional feature tensor. S26. Fill the empty voxel grid with zero values ​​and obtain the corresponding tensor; S27. Combine the tensors obtained in steps S25 and S26 to generate multidimensional BEV pseudo-image features.

4. The intelligent traffic target detection system based on 4D millimeter-wave and lidar point cloud fusion as described in claim 1, characterized in that: The processing sub-steps in the sparse mesh attention module are as follows: S31. Based on the BEV pseudo-image features obtained in the point cloud bidirectional fusion coding module, generate a sparse mask matrix. S32. Flatten the mask and obtain N non-empty grid features according to the index, construct a compact feature sequence, and use this as the valid grid token; S33. Input the compact feature sequence obtained in step S32 into the self-attention layer, and use a multilayer perceptron to map the channel of dimension C to the attention space of hidden dimension D, and output the compact feature sequence of dimension D. S34. Perform LayerNorm operation on the compact feature sequence in dimension D obtained in step S33. The results are linearly mapped through three independent multilayer perceptrons to obtain query vector, key vector and value vector. S35. Based on the vector obtained in step S34, calculate the sparse attention; S36. The sparse attention obtained in step S35 and the compact feature sequence obtained in step S33 are fed to the feedforward network and the residuals are added again to obtain the enhanced features. S37. The channel features of dimension D are projected onto the original space of dimension C through a multilayer perceptron to obtain the updated compact feature sequence. S38. Using the grid index recorded in step S32, restore the updated compact feature sequence to a pseudo-image grid of size H×W, thereby obtaining a new pseudo-image feature map.

5. The intelligent traffic target detection system based on 4D millimeter-wave and lidar point cloud fusion as described in claim 1, characterized in that: The multi-scale cross-attention fusion network module includes: The multi-scale feature extraction module uses a three-layer convolutional network to downsample the features of the two-modal pseudo-images to different scales to obtain multi-scale features. The cross-attention feature fusion module performs feature interaction on the same scale features of two modalities among the obtained multi-scale features. Each scale uses a cross-attention unit to process and calculate to generate highly separable feature vectors at specific locations. The feature upsampling fusion module upsamples the obtained multi-scale feature maps to the same spatial resolution. At the same time, it aligns features of different scales to the same scale to facilitate subsequent fusion. The multi-scale feature maps are stitched together along the channel dimension to finally form the fused feature map that serves as the input to the detection head.

6. The intelligent traffic target detection system based on 4D millimeter-wave and lidar point cloud fusion as described in claim 5, characterized in that: The cross-attention unit processing calculation process is as follows: S421. Perform dimension alignment and dimensionality reduction operations on LiDAR features and 4D millimeter-wave radar features through two convolutional layers with the same kernel size, while preserving key semantic information to generate query vectors and key features. S422. The obtained dimensionality reduction features are used to construct a spatial correlation matrix through matrix dot product operation and normalized by the softmax function to characterize the similarity measurement between different modal features at each spatial location. Simultaneously, the context information of the LiDAR features is aggregated based on the convolutional layer. S423. Obtain lidar enhancement features about the feature map location by weighted aggregation of 4D millimeter-wave radar feature context information; S424. The 4D millimeter-wave radar enhancement feature is generated by processing steps S421-S423. S425. By splicing enhanced features of different scales according to the channel dimension and adjusting the dimension through the convolutional layer of the convolutional kernel, scale fusion features are obtained.

7. The detection method of the intelligent transportation target detection system based on 4D millimeter wave and lidar point cloud fusion according to any one of claims 1-6, characterized in that: S1. Extract target motion features based on dynamic feature enhancement module, expand the radial velocity obtained by point cloud compensation, and construct a point cloud motion state prediction network focusing on velocity features. S2. Embed the point cloud features of 4D millimeter wave and lidar to bridge the differences in heterogeneous modal data. At the same time, extract voxel-level features using column voxels as units and project them onto a 2D plane to obtain BEV pseudo-images. S3. Use sparse masking to perform attention calculation only on non-empty grids, extract 4D millimeter-wave radar pseudo-image features, and obtain new pseudo-image features. The sparsity construction based on BEV features can effectively extract sparse mesh attention modules for this feature. S4. Multi-scale multimodal features are fused through a multi-scale cross-attention fusion network, thereby integrating complementary information from different granular levels of multimodal features to form a fused feature map that can be used as input to the detection head. S5. Input the obtained fused feature map into the detection head to obtain high-confidence detection of 3D targets.

Images