A multi-source heterogeneous sensor information fusion target detection and tracking network model and method

Abstract

The application discloses a multi-source heterogeneous sensing information fusion target detection and tracking network model and method. In recent years, the intelligent automobile environment perception technology has made significant progress, however, the wide deployment of unmanned vehicles has not been realized, the main restricting factor lies in the problem of inaccurate driving environment state estimation in complex traffic scenes. Therefore, the application proposes a method for efficiently fusing multi-source heterogeneous sensing information, constructs a camera view conversion module guided by accurate spatial information, and designs a query initialization module with prior information. In addition, the application adopts a query enhancement module based on timing information, improves the detection accuracy through the tracking task, combines the data association matching strategy of LSTM and Transformer, and realizes multi-target tracking. After training by 8*Tesla V100GPUs, the application significantly improves the accuracy of the detection and tracking algorithm on the nuScenes and RADIATE data sets, and provides a high-precision, high-reliability and high-safety technical solution for autonomous driving.

Description

Technical Field

[0001] This invention belongs to the field of environmental perception technology for intelligent autonomous vehicles, specifically referring to a target detection and tracking network model and method that integrates multi-source heterogeneous sensor information. Background Technology

[0002] Intelligent autonomous vehicles mainly consist of three systems: environmental perception, path planning, and decision control. Intelligent autonomous vehicles utilize massive amounts of data, such as raw camera images and radar point clouds, provided by various onboard environmental perception sensors. The environmental perception system needs to accurately and reliably extract useful information for driving from this vast amount of raw data, forming the foundation of the autonomous vehicle system and ensuring safe and reliable vehicle operation. To date, most existing 3D object detection and tracking methods are based on common driving scenarios provided by mainstream datasets. Research on real-world complex driving scenarios (strong light, darkness, rain, fog, occlusion, etc.) has been neglected. Ensuring that intelligent vehicles can robustly and accurately detect and track objects in various environments is crucial for driving safety, and this is one of the bottlenecks that has prevented autonomous vehicles from reaching Level 4 or higher levels of autonomous driving for a long time.

[0003] To address complex driving scenarios, the development of in-vehicle environmental perception systems has trended towards the concept of multi-source heterogeneous sensor information fusion. This combines the advantages of three environmental perception sensors—visual cameras, LiDAR, and millimeter-wave radar—to achieve comprehensive, three-dimensional, and high-precision environmental perception, contributing to a higher level of autonomous driving capabilities. This multi-source heterogeneous sensor information fusion approach leverages the strengths of each sensor to overcome the limitations of single-sensor devices. For example, visual cameras lose detection capabilities in darkness or bright light, failing to perceive depth information; LiDAR performance degrades in adverse weather conditions (such as rain, snow, and fog); and millimeter-wave radar has lower resolution and accuracy. Simultaneously, multi-sensor fusion improves the redundancy and robustness of the perception system, resulting in superior performance in handling abnormal situations. For future intelligent vehicles, only by achieving multimodal environmental perception and stable information processing can the requirements for safe driving in various complex environments be met.

[0004] 3D object detection based on multi-sensor fusion: Bird's Eye View (BEV). The main difference between different methods lies in the method of acquiring image BEV and the multimodal fusion approach. TransFusion compresses image features in the height direction to obtain image BEV features and uses a two-layer Transformer decoder for adaptive fusion of image and LiDAR point cloud. Multi-view features are rendered into the BEV feature space. The computational cost is proportional to the scale of the 3D space, so this method inevitably cannot be scaled to large-scale scenes. BEVFusion directly concatenates the image BEV features with the LiDAR point cloud BEV features. However, the high-dimensional heterogeneous features fused in this way may interfere with each other, which may make it difficult for the network to understand. The OpenOccupancy method uses occupancy probability to describe the 3D scene, which can more comprehensively describe the real driving scene and effectively solve the long-tail problem, but the computational cost is huge and difficult to implement in real vehicles. In contrast, cross-attention shows more advantages in processing multimodal features. Therefore, this method has been widely used in some recent studies (such as CMT, MixedFusion, Uniad, ReasonNet, and Interfuser). However, these methods initialize the learnable query for cross-attention with randomly generated parameters, failing to utilize the prior information inherent in multimodal features. This can also lead to misalignment of the same key object across multiple modalities, ultimately resulting in slower and less optimized model learning convergence.

[0005] Currently, there are two main tracking frameworks for multi-object tracking (MOT): detection-based tracking frameworks and joint detection tracking frameworks. The main idea of ​​detection-based tracking frameworks is to perform state prediction, data association, and trajectory management tasks accordingly, and then output tracking results. CenterPoint-Enhance, proposed by Wang Hai et al., employs a state prediction module based on Kalman filtering and neural networks, and a two-stage data association strategy to improve tracking accuracy. CAMO-MOT uses cascaded matching technology to design a spatial dynamic information association system. In its first-level matching stage, spatial motion information is widely used for correlation identification of most objects. Then, in the second-level matching stage, image appearance features are invoked to associate the remaining objects. This matching technology can establish and maintain object associations in multiple dimensions, thereby achieving behavior analysis across time and space. This characteristic is particularly crucial when dealing with various situations that may arise in complex scenes. EagerMOT employs a two-stage association algorithm. In this data association framework, the first stage is standard Hungarian data association, which targets the identification and tracking of 3D bounding boxes. The joint detection and tracking framework combines detection and tracking into a single, end-to-end trainable network. In the MotionTrack algorithm, the Transformer architecture is used for data association, enabling the output of detection results while completing multi-target tracking tasks.

[0006] Although existing 3D target detection and tracking algorithms have made some progress, their performance in complex environments is still unsatisfactory. After extensive research and discussion, current target detection and tracking methods based on multi-source heterogeneous sensor information fusion have the following shortcomings:

[0007] (1) Existing methods are all trained and tested based on ordinary driving scenarios provided by mainstream datasets, but the research on complex environments often encountered in actual driving is neglected. Due to factors such as the mixing of motor vehicles and non-motor vehicles in complex traffic scenarios, the randomness and uncertainty of severe weather, and the uncertainty of multi-source sensor data fusion, existing methods are difficult to meet the requirements of high accuracy, high reliability, and high safety in complex environments.

[0008] (2) Millimeter-wave radar point clouds have strong penetration capabilities and their performance is almost unaffected by rain and fog, which is an advantage that other sensors cannot match. However, in the current multimodal fusion framework, millimeter-wave radar generates radar point cloud data with low resolution and serious false detection. Simply introducing millimeter-wave radar point clouds into data fusion not only fails to improve the robustness of the algorithm, but also reduces the confidence of the fused data.

[0009] (3) Although the LSS algorithm used for camera view transformation is very mature, its perspective imaging principle inevitably causes the loss of depth information, which in turn affects the accuracy of image branch detection and the final fusion detection result.

[0010] (4) Classical algorithms such as Anchor-DETR and PETR use randomly generated parameters to initialize learnable queries, and then the object queries are optimized through multiple Transformer decoder layers. However, if the initialized object queries have multimodal dense prior information, the difficulty of the model in the optimization process can be reduced while improving the overall performance. Summary of the Invention

[0011] To address the aforementioned issues and improve the accuracy, reliability, and driving safety of intelligent vehicles in complex traffic scenarios, this invention proposes a robust target detection and tracking network model for complex traffic scenarios: RobustFusion, which integrates multi-source heterogeneous sensor information. This network model includes a multimodal feature extraction and BEV feature generation module, a LiDAR-guided camera view transformation module, a query initialization module with prior information, a feature fusion module based on a cross-modal attention mechanism, and a data association matching module based on LSTM and Transformer. The implementation of this invention includes the following steps:

[0012] S1: Design a multimodal feature extraction and BEV feature generation module

[0013] This invention designs a network model called RobustFusion, which employs an improved feature extraction framework based on the BEVFusion multi-branch architecture. This framework effectively integrates multimodal sensor data to achieve advanced feature representation. The feature extraction networks of this network model are somewhat independent yet interconnected, processing camera images, LiDAR, and millimeter-wave radar point cloud data separately. For the lidar point cloud processing branch and the millimeter-wave radar point cloud processing branch, this invention directly extracts BEV features. For the camera image processing branch, in order to obtain the BEV features of the camera image more accurately, a camera view transformation module based on accurate spatial information is constructed. In addition, millimeter-wave radar is often rarely used due to the low resolution and serious false detection of the radar point cloud data it generates. However, millimeter-wave radar has strong penetration ability and its performance is almost unaffected in rainy and foggy weather. Its advantage in rainy and foggy weather is unmatched by other sensors. In order to make full use of the advantages of millimeter-wave radar in complex environments, this invention introduces a backbone network based on a neighborhood attention mechanism into the millimeter-wave radar point cloud processing branch, which enhances the model's ability to capture key information and improves the understanding of the environment by millimeter-wave radar point cloud data.

[0014] S2: Design a LiDAR-guided camera view transformation module

[0015] The LSS algorithm used for camera view transformation inevitably suffers from a lack of depth information due to its perspective imaging principle, which in turn affects the accuracy of image branch detection and the final fusion detection result. To better utilize the accurate spatial information provided by LiDAR point clouds, a LiDAR-guided camera view transformation module is constructed. This module uses LiDAR as a priori guidance to obtain an accurate image representation in the BEV space. This module incorporates the BEV features from the previous frame... BEV characteristics of the current frame lidar The image features are fused into query features, which act as the key and value in the attention mechanism. The fused query features and image features undergo deformable cross attention to update the camera BEV features.

[0016] S3: Construct a query initialization module with prior information

[0017] Classic algorithms like Anchor-DETR and PETR initialize learnable queries using randomly generated parameters, followed by optimization of object queries through multiple Transformer decoder layers. If the initialized object query contains multimodal, dense prior information, it can reduce the difficulty of the model's optimization process and improve overall performance. Therefore, this invention constructs a query initialization scheme with prior information to fully utilize multimodal data features. This invention also employs a query enhancement module based on temporal information to enhance detection accuracy using tracking tasks. Specifically, the query initialization scheme with prior information in this invention involves using trimodal BEV features for object query initialization. The trimodal BEV features are simultaneously input into an occupancy prediction network to generate a heatmap, which is then encoded into position and content embeddings via a feedforward network. Furthermore, this invention introduces a query enhancement module based on temporal information, enabling the model to utilize tracking tasks to improve detection accuracy. Step S5 will elaborate on the temporal information-based query enhancement module.

[0018] S4: Design a feature fusion module based on cross-modal attention mechanism

[0019] This invention extends deformable attention to the multimodal domain and designs a cross-modal deformable attention mechanism. The query with prior information generated in step S3 will interact with the multimodal fusion features. By using the deformable attention mechanism for cross-modal feature fusion, it is possible to adaptively weight and sum the sampled features from various modal information to fuse multimodal features.

[0020] S5: Design of Data Association Matching Based on LSTM and Transformer

[0021] This invention proposes a data association matching module based on LSTM (Long Short-Term Memory) and Transformer. This module includes three key steps: LSTM-based query feature update, target feature update, and association between query and target features.

[0022] The present invention also designs a query enhancement module based on time-series information, which aims to construct and enhance the object features detected in previous frames. The enhanced results will be applied to step S3 to enhance detection accuracy by using the tracking task.

[0023] S6: Model Training and Testing

[0024] This invention constructs a network model for application in the field of computer vision, particularly aimed at enhancing the perception capabilities of autonomous driving systems under various adverse weather conditions. To achieve the desired results, the proposed network model was trained on the RADIATE dataset, which covers a variety of extreme weather environments, enhancing the model's environmental adaptability. The fusion of three perception sensors in the dataset provides the model with stereo images, LiDAR, and millimeter-wave radar data, even though these may suffer from signal attenuation or noise interference in some complex scenarios.

[0025] The proposed network model significantly outperforms the benchmark algorithm in terms of accuracy. The RobustFusion network achieves a detection accuracy (mAP) of 70.1% in complex traffic scenarios and a multi-target tracking accuracy (AMOTA) of 63.3%, surpassing the benchmark algorithm's 65.7% mAP and 60.6% AMOTA.

[0026] The performance advantages of RobustFusion under complex environmental conditions were demonstrated through visualization. Furthermore, testing on the real-world nuScenes dataset also confirmed its excellent performance and practicality.

[0027] S7: Deploy the trained network model to intelligent driving vehicles; conduct real-vehicle deployment and road tests based on the ROS system to achieve real-time target detection and tracking.

[0028] In summary, the RobustFusion network model proposed in this invention promotes the stability and accuracy of autonomous driving under different environmental conditions, significantly improves the performance of perception systems under adverse weather conditions, and contributes a highly robust technical solution to the field of autonomous driving.

[0029] The beneficial effects of this invention are:

[0030] 1. This invention efficiently integrates multi-source heterogeneous sensing information, effectively improving the robustness of target detection and tracking algorithms in complex environments such as rain, snow, and fog, and reducing the impact of environmental noise on environmental perception results.

[0031] 2. Construct a camera view transformation module based on accurate spatial information to obtain BEV features of camera images more accurately.

[0032] 3. Construct a query initialization scheme with prior information, adopt a query enhancement module based on time-series information, and enhance detection accuracy by using tracking tasks.

[0033] 4. This is the first time that data association matching combining LSTM and Transformer has been used to track multiple types of targets, and there is no explicit state estimation in the tracking algorithm.

[0034] 5. After training and testing with 8 Tesla V100 GPUs, this invention significantly improves the accuracy of detection and tracking algorithms on the nuScenes and RADIATE datasets. RobustFusion achieves a detection accuracy mAP of 70.1% and a multi-target tracking accuracy AMOTA of 63.3% in complex environments such as night and rain in nuScenes, while the benchmark algorithm BEVFusion achieves an mAP of 65.7% and an AMOTA of 60.6%. This significantly improves the accuracy of detection and tracking algorithms under adverse weather conditions, providing a high-precision, high-reliability, and high-safety technical solution for autonomous driving. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the implementation process of the present invention;

[0036] Figure 2 This is the overall network architecture of the RobustFusion detection algorithm of this invention;

[0037] Figure 3 This is the network architecture of the LiDAR-guided camera view conversion module of the present invention;

[0038] Figure 4 This is a scene visualization from the RADIATE dataset used in this invention;

[0039] Figure 5 This is a visualization of the object detection results of this invention on the nuScenes dataset;

[0040] Figure 6 This is a visualization of the target detection and multi-target tracking results of the present invention;

[0041] Figure 7This is the pseudocode of the network model algorithm proposed in this invention;

[0042] Figure 8 This invention uses the Jiangda Intelligent Driving No. 2 experimental platform for unmanned vehicles. Detailed Implementation

[0043] The invention will now be further described with reference to the accompanying drawings.

[0044] The implementation flowchart of the present invention is as follows: Figure 1 As shown, the specific implementation includes: selecting the severe weather datasets RADIATE and nuScenes as training and validation sets for the network model; designing object detection algorithms, including a LiDAR-guided camera view transformation module, a query initialization module with prior information, and a feature fusion module based on cross-modal attention mechanism; designing a multi-object tracking algorithm based on LSTM and Transformer data association matching; training the object detection and tracking network using the training set; and finally, deploying the trained optimal network model on a real vehicle to achieve highly robust road object detection and tracking. The specific network architecture of RobustFusion is as follows: Figure 2 As shown. The implementation of this invention includes the following steps:

[0045] Step 1: Extract BEV features from camera images, LiDAR point clouds, and millimeter-wave radar point clouds.

[0046] In the feature extraction stage, this invention employs an improved BEVFusion multi-branch architecture to extract multimodal features. Specifically, the camera image, LiDAR point cloud, and millimeter-wave radar point cloud of the current frame are input into three interconnected but relatively independent backbone networks to form a higher-level feature representation.

[0047] The proposed RobustFusion network model first extracts the BEV features of the current camera image, LiDAR point cloud, and millimeter-wave radar point cloud. In the camera image processing branch, the mature image feature extraction network VoVNet is first used to extract image features, and its spatial dimension is given by formula (1). Other pre-trained frameworks, such as Swin Transformer and ResNet, can also be used for image feature extraction.

[0048]

[0049] in, This is a feature tensor representing an intermediate process in the image processing branch. C, H, and W represent the number of channels, image height, and image width of the feature tensor, respectively. It's important to note that here... It is not the final BEV feature of the camera, but the image feature with dimension of formula (1).

[0050] Secondly, the generation of image BEV features requires the LiDAR-guided camera view transformation module in step two.

[0051] For the LiDAR point cloud processing branch, the input point cloud is first voxelized. The pre-trained VoxelNet network is used to voxelize the original LiDAR point cloud, converting it into a uniform mesh. Simultaneously, to improve computational efficiency, a 3D sparse convolutional SECOND network is used to encode features in the voxel space, forming a feature representation in the BEV space, with the following spatial dimensions:

[0052]

[0053] in, It is the feature tensor of the LiDAR point cloud processing branch in the BEV space, where C, X, and Y represent the number of channels, the dimension of the feature tensor in the X-axis direction, and the dimension of the feature tensor in the Y-axis direction, respectively.

[0054] In current multimodal fusion frameworks, millimeter-wave radar is often less used due to the inherent sparsity of its generated radar point cloud data. To overcome this obstacle and effectively utilize the potential of millimeter-wave radar in autonomous driving and advanced driver assistance systems (ADAS), this invention proposes an innovative data preprocessing and feature extraction method aimed at improving the semantic parsing capabilities of sparse millimeter-wave radar point cloud data. First, this invention employs columnar processing as a preprocessing step, transforming the raw millimeter-wave radar point cloud data into a more standardized and structured representation. This transformation preserves the inherent structural characteristics of the data and creates conditions for subsequent deep learning model processing. Furthermore, to better extract features from the bird's-eye view (BEV) point cloud data of millimeter-wave radar, this invention introduces a network architecture based on a neighborhood attention mechanism as the backbone network. This attention mechanism can effectively identify key structures and patterns in the point cloud data, and by weighting local neighborhood features, it strengthens the model's ability to capture valuable information from the millimeter-wave radar point cloud. Furthermore, through a neighborhood attention mechanism, the model can accurately extract semantic-level features from the columnarized point cloud data, significantly enhancing the environmental descriptive capabilities of millimeter-wave radar data. The spatial dimension of the BEV point cloud features from millimeter-wave radar is:

[0055]

[0056] The above process can be represented as:

[0057]

[0058]

[0059]

[0060] in, Indicates a cascading operation. This represents point cloud input. Indicates camera image input, This indicates the BEV characteristic of the lidar. Representing image features, , These represent the voxelization and 3D sparse convolution operations of the lidar point cloud, respectively. This indicates that an image feature extraction network is used to extract image features. The attention mechanism for representing millimeter-wave radar point cloud columnarization and neighborhood attention is used. express.

[0061] Step 2: Construct a LiDAR-guided camera view transformation module

[0062] LiDAR and image extraction provide 3D geometric and semantic information, respectively. To efficiently fuse features extracted from different modalities, these features must first be merged into a unified BEV space. This requires view transformation to project the 2D image features into the camera's BEV space. Projecting the image into 3D space typically uses the LSS algorithm to project multi-view images into the BEV space. Here, the Lift operation predicts the discrete depth distribution of the features, converting it into frustum features, and then a frustum pooler is used to compress the height information. While the LSS algorithm offers numerous advantages in 3D scene reconstruction and autonomous driving applications, it suffers from issues with accuracy and resolution. The estimated depth may not be accurate enough, and it is susceptible to factors such as lighting variations and occlusion.

[0063] To address the aforementioned issues, this invention employs a camera view transformation guided by LiDAR information, relying on the relatively accurate spatial information of the pre-trained LiDAR BEV (Browser Image Value) to acquire corresponding semantic features in the image. The LiDAR-guided camera view transformation module proposed in this invention can effectively project two-dimensional image features into the BEV space.

[0064] LiDAR-guided camera view transformation module: Utilizing a deformable attention module, 2D image features are transformed into BEV features under the guidance of accurate spatial priors generated by the LiDAR. Figure 3 The network architecture of the LiDAR-guided camera view transformation module is shown.

[0065] This module will use the camera's BEV features from the previous frame. BEV characteristics of the current frame lidar The image features are fused together to form query features, acting as both keys and values ​​in the attention mechanism. Deformable cross-attention is performed with the image features to update the camera BEV features. The LiDAR-guided camera view transformation module can be described as follows:

[0066]

[0067]

[0068] in, The precise spatial prior generated by the current frame's LiDAR BEV features and the previous frame's camera BEV features serves as the query in the attention mechanism. It's a connection operation. It is the BEV feature of the final generated camera image. It is a deformable attention module. It is the image feature of the current frame.

[0069] Step 3: Construct a query initialization module with prior information

[0070] Unlike the DETR model in existing technologies, the strategy adopted in this invention no longer relies on randomly generated parameters to initialize learnable queries. Instead, this invention introduces encoding with prior information obtained from multimodal features. This means that RobustFusion can leverage rich environmental information from multiple sensing sensors, thereby providing stronger semantic and contextual information support for the subsequent object detection fusion module based on a deformable attention mechanism.

[0071] This invention proposes a query initialization scheme with prior information, aiming to improve the initialization accuracy of object queries in autonomous driving scenarios, thereby generating a richer and more comprehensive environmental representation. This invention extracts BEV features from data acquired by three different sensors—visual cameras, LiDAR, and millimeter-wave radar—through steps one and two. In detail, the core of this part utilizes an occupancy prediction network, which maps the trimodal input BEV features to a high-dimensional heatmap. This heatmap spatially represents the possible locations of the detected objects, providing a rough object location information. This step achieves the transformation from data from the three sensors to a unified environmental representation, taking into account the advantages and characteristics of each sensor and effectively combining their different spatial perceptions. Furthermore, the heatmap is processed by a feedforward network, which encodes information from the high-dimensional space into a more compact space, thus completing location encoding and content encoding. Location encoding captures the spatial location information of the object, while content encoding involves more about the object's feature description. This not only optimizes the object query initialization process but also provides a solid foundation for subsequent tasks such as object detection, classification, and tracking through precise location and content embedding. Furthermore, the combined use of occupancy prediction and feedforward networks further reduces uncertainties caused by sensor noise and data inconsistencies, significantly improving the system's adaptability and robustness in complex and dynamic environments.

[0072] In addition, a query enhancement module based on time-series information is introduced, enabling the model to leverage tracking tasks to improve detection accuracy. Step five will elaborate on the query enhancement module based on time-series information. The above can be expressed by the following formula:

[0073]

[0074]

[0075]

[0076] in, Indicates the query code. Indicates position code, Indicates content encoding. This indicates enhanced timing information. Indicates a feedforward network. Indicates a connection operation. Represents the BEV features of the image. This represents the BEV feature of the lidar point cloud. This represents the BEV characteristics of millimeter-wave radar point clouds.

[0077] Step 4: Construct a feature fusion module based on cross-modal attention mechanism

[0078] Deformable attention models have achieved significant progress in computer vision tasks such as image recognition and object detection due to their excellent ability to align target and background features. Extending the deformable attention mechanism to multimodal scenarios is a natural and beneficial exploration. The core contribution of this invention lies in proposing a deformable attention mechanism that fuses multimodal information. By introducing this mechanism into the traditional cross-modal information fusion process, this invention creatively extends the capabilities of deformable attention models to multimodal data processing, particularly for visual images, LiDAR point clouds, and millimeter-wave radar signals. Specifically, this model achieves cross-modal feature fusion through the following steps: Unlike classic algorithms such as Anchor-DETR and PETR, which use randomly generated parameters to initialize learnable queries, this system no longer relies on random parameters to initialize learnable queries. Instead, the network model constructs queries in a bottom-up manner using information extracted from multimodal features. After the query initialization process in step three, the query becomes more informative because it already contains rich semantic and contextual information encoded by sensor data. The attention mechanism in this invention considers not only spatial and feature transformations when processing multimodal inputs, but also the differences between different modalities. By predicting the attention center and shape, the model can adaptively switch and adjust between different modalities, achieving more accurate feature fusion. Specifically, in processing multimodal data, the deformable attention mechanism first extracts features from visual, LiDAR, and millimeter-wave radar signals respectively, and then introduces dynamic weights to adaptively fuse these features. In this process, the deformable attention mechanism adaptively assigns weights to features from different modalities, and then fuses the features by summing the weights. This method allows the model to discover and utilize the inherent correlations between modalities without explicit guidance.

[0079] (12)

[0080] In the above formula, first, F j Embed the three learnable encodings obtained in step three. Then, F i Convert to query z qi and reference point p qi and F j Consider them as keys and values. F i F j Representing image features, LiDAR, and millimeter-wave radar point cloud features respectively, variable m is used to index the attention heads, M represents the number of attention heads, variable k is used to index the sampling keys, K represents the number of sampling keys, and ∆Pmqk Indicates the sampling offset, A mqk This represents the attention weights. This formula updates F through a deformable cross-attention mechanism. j By exchanging F i and F j The three features represented are then updated through six different feasible variable cross-attention mechanisms to update the point cloud features of image features, lidar, and millimeter-wave radar.

[0081] Finally, two feedforward networks are used to predict the 3D bounding boxes of the detected targets using the updated query. and category information The prediction process is described as follows:

[0082]

[0083] in, and These represent the feedforward networks for regression and classification tasks, respectively, and Query is the updated object query.

[0084] Then, binary matching is applied to a one-to-one assignment between predictions and true values, focus loss is used for classification, and L1 loss is used for 3D bounding box regression:

[0085]

[0086] in, Represents the total loss function. It is a classification loss function that calculates the predicted classification information. With real classified information The losses between This represents the regression loss function, used to calculate the predicted bounding box information. Compared with the actual bounding box information The losses between and These represent the weight hyperparameters of the classification loss term and the weight hyperparameters of the regression loss term, respectively.

[0087] Step 5: Data association matching based on LSTM and Transformer

[0088] This invention proposes a data association and matching module based on LSTM (Long Short-Term Memory) and Transformer to achieve target tracking. This module comprises three parts: LSTM-based query feature update, target feature update, and association between query and target features. The specific process is as follows:

[0089] This invention introduces an LSTM module into the Transformer-based data association and matching module. The purpose is to learn from time series data by inputting the position coordinates of objects in consecutive frames, previously calculated motion indicators such as velocity and acceleration, and thus help the LSTM model identify the motion trajectory of the objects and estimate their heading angle.

[0090] The query features and inputs from the previous frame's detection task, along with the LSTM's output heading angle, are used as inputs. Two cross-attention layers are then used to update the features of the detected objects. The data association process consists of two layers. The first layer updates the appearance features of the detected objects based on appearance feature data association, as data association based on the similarity between appearance features is crucial. The second layer is based on motion feature data association, aiming to inherit the object's motion features. Since most single-frame detection algorithms cannot accurately estimate object motion (such as velocity and acceleration), the LSTM's output heading angle becomes the most critical motion feature introduced into the data association process.

[0091] The goal of the target feature update module is to refine the candidate features of objects in the current frame to facilitate data association. This module simply receives the query input (Q-in) of the current frame and processes it through two layers of multilayer perceptron (MLP), without updating the features of the current frame.

[0092] Since objects detected in the previous frame may disappear in the current frame, a zero-padded empty vector (i.e., the "dead query feature") is concatenated with Q-in to represent queries for disappeared objects. Finally, a dot product operation is used to associate the updated query features from the previous frame with the refined target features of the current frame. The output of the dot product operation is an N * (M + 1) matrix, where N represents the number of objects detected in the previous frame and M represents the number of object queries in the current frame. Higher values ​​in the matrix (association scores) indicate a greater likelihood of association. During the training phase, this invention treats the association process as a classification task and uses the cross-entropy loss function to calculate the loss between the estimated results and the true results of the object association module.

[0093] Finally, in the evaluation phase, a greedy search-based method is applied after object association to prevent duplicate associations.

[0094] However, in object detection algorithms, although query initialization guided by prior information can provide rich contextual semantic information, it cannot fully utilize the continuity of the environmental sequence and the dynamic information of object motion if the introduction of temporal information is not taken into account.

[0095] To address this issue, this invention further proposes a query enhancement module based on temporal information. This module can effectively model the motion trend of objects and is combined with query initialization guided by prior information. Specifically, by effectively modeling the perceptual information of consecutive frames, the motion trajectory of objects can be predicted and tracked more accurately, thereby enhancing detection accuracy through tracking tasks. The specific implementation process is as follows: In the query enhancement module based on temporal information, the query from the previous frame is... And querying the current frame The query from the previous frame is passed to the cross-attention mechanism. As a query in this attention mechanism, it aggregates features from the previous frame into the corresponding query of the current frame. In this way, historical frame information can be effectively aggregated into the current frame, and for newly generated... This will be directly used as the query input for the cross-modal attention mechanism in the detection phase.

[0096] Step Six: Model Training and Testing

[0097] Steps one through five constitute the network model of this invention. The pseudocode for the network model algorithm is as follows: Figure 7 As shown, this network is trained end-to-end. The RADIATE dataset provides six driving scenarios, including normal urban driving, high-exposure driving, foggy driving, nighttime driving, snowy driving, and rainy driving, encompassing a large number of highly challenging and complex environments. This dataset includes three environmental perception sensors: stereo cameras (providing images with a resolution of 672 × 376, 15 frames per second per camera); LiDAR (using a 32-channel, 10Hz LiDAR to achieve 360° coverage; however, in complex environments, the LiDAR signal may be severely attenuated and reflected by dense fog or snow, resulting in missing, noisy, and inaccurate data); and millimeter-wave radar (providing 360° high-resolution ranging and azimuth images; its maximum operating range is 100 meters, with a ranging resolution of 0.175 meters, an azimuth resolution of 1.8°, and an elevation resolution of 1.8°).

[0098] During training, the input image is randomly cropped to 256×256 pixels, and image data augmentation is performed using methods such as random erasure and horizontal flipping. The network model is implemented on the PyTorch platform and runs on eight Tesla V100 GPUs with 32GB of dedicated memory each. The model is trained for 50 epochs using the Adam optimizer, with the learning rate varying using a cosine fire method, and a maximum learning rate of 0.005.

[0099] RobustFusion achieved a detection accuracy (mAP) of 70.1% and a multi-target tracking accuracy (AMOTA) of 63.3% in complex environments such as night and rain in nuScenes, while the benchmark algorithm BEVFusion achieved an mAP of 65.7% and an AMOTA of 60.6%, significantly improving the accuracy of detection and tracking algorithms under adverse weather conditions.

[0100] The visualization results of the network model proposed in this invention tested on the nuScenes dataset in a real-world scenario are shown in the figure below. Figure 5 As shown, the top six groups are the prediction results of the network model proposed in this invention, and the bottom six groups are the actual ground truth bounding boxes. The visualization shows that RobustFusion significantly improves the detection accuracy under severe weather conditions.

[0101] To more vividly demonstrate the advantages of this invention, such as Figure 4 As shown, the object detection and multi-object tracking results of three representative complex environments from the RADIATE dataset are visualized. The specific results are as follows: Figure 6 The images show driving conditions under high exposure, rain / fog, and snow conditions. The first row displays the original input image from the camera and the detection results generated solely by camera image methods. The second row shows the detection results generated solely by millimeter-wave radar point cloud methods and lidar point cloud methods. The third row shows the tracking results generated by RobustFusion. White numbers represent trajectory IDs, and red dashed lines indicate missed detections in single-sensor detection. These visualizations demonstrate that RobustFusion significantly improves the accuracy of detection and tracking algorithms under adverse weather conditions.

[0102] Step 7: Deploy the trained network model to the vehicle to achieve real-time target detection and tracking for intelligent driving.

[0103] For real-vehicle testing, the trained optimal network model can be deployed in real vehicles to achieve highly robust road target detection and tracking. The real-vehicle road test is based on the "Jiangda Intelligent Driving-2" unmanned driving experimental platform, which is modified from a Chery Arrizo-5e pure electric vehicle. Figure 8As shown, this experimental platform adds an environmental perception system and a vehicle positioning system to the original vehicle, and modifies the vehicle's control and execution mechanisms. Furthermore, an industrial control computer and a vehicle controller are installed inside the vehicle to meet the needs of autonomous driving experiments. For the environmental perception system, sensors such as LiDAR, cameras, and millimeter-wave radar are added, and these sensors are jointly calibrated to perceive the environmental information around the vehicle in autonomous driving scenarios. Regarding the intelligent vehicle positioning mechanism, the platform is equipped with a Global Positioning System (GPS), gyroscopes, and inertial measurement units to accurately determine the vehicle's geographical location. Improvements to the vehicle's control and execution units mainly include the reconfiguration of the vehicle's chassis system and electronic control equipment, such as implementing electronic control of the throttle, transmission, steering, and braking systems, thereby achieving precise control of the vehicle's lateral movement and longitudinal acceleration. After modification, the experimental platform gains the ability to integrate multi-sensor data fusion, high-precision pathfinding, decision-making, and complete vehicle control.

[0104] The algorithm was deployed on a real-world autonomous driving platform based on the ROS system, and daytime and nighttime experiments were conducted in Zhenjiang city, on elevated roads, and on highways. The real-world road tests showed that the RoubustFusion algorithm can detect vehicles, pedestrians, and cyclists under different time and road conditions, with good detection performance, meeting the detection needs of autonomous vehicles in complex traffic scenarios.

[0105] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. A target detection and tracking system that fuses multi-source heterogeneous sensor information, characterized in that, include: Multimodal feature extraction and BEV feature generation module, LiDAR-guided camera view transformation module, query initialization module with prior information, feature fusion module based on cross-modal attention mechanism, and data association matching module based on LSTM and Transformer; The multimodal feature extraction and BEV feature generation module processes camera images, LiDAR point clouds, and millimeter-wave radar point cloud data to generate BEV features. Specifically, for LiDAR point clouds and millimeter-wave radar point clouds, the LiDAR BEV features and millimeter-wave radar BEV features are directly extracted. For camera image processing, this module is responsible for extracting image features. The lidar-guided camera view transformation module: using lidar as a priori guide, and utilizing a deformable attention module to transform two-dimensional image features into camera BEV features under the guidance of the precise spatial prior generated by lidar. The query initialization module with prior information uses three BEV features for object query initialization. The three BEV features are simultaneously input into the occupancy prediction network to generate a heatmap, realizing the transformation of three types of multimodal data into a unified representation. Then, the data is encoded by a feedforward network to generate location embeddings and content embeddings. The location embedding is responsible for capturing the spatial location information of the object, and the content embedding is used for the feature description of the object. Finally, the query code is generated by combining the temporal information enhancement module based on temporal information. The feature fusion module based on the cross-modal attention mechanism: uses the query encoding with prior information generated by the query initialization module with prior information to interact with the multimodal fusion features. By adopting a deformable attention mechanism for cross-modal feature fusion, the sampled features from each modality are adaptively weighted and summed to achieve the fusion of multimodal features. The data association and matching module based on LSTM and Transformer achieves target tracking through LSTM-based query feature updates, target feature updates, and the association between query and target features. The data association and matching module based on LSTM and Transformer processes the following steps: The algorithm uses the query features, query input, and heading angle output from the LSTM from the previous frame's detection task as input, and updates the features of the detected objects through two cross-attention layers. The data association process is divided into two layers: the first layer is based on appearance features, updating the appearance features of the detected objects; the second layer is based on motion features. Since most single-frame detection algorithms cannot accurately estimate object motion, the heading angle output from the LSTM becomes the most critical motion feature introduced into the data association process. The goal of target feature update is to refine the candidate features of the objects in the current frame to facilitate data association. The target feature update part receives the query input Q-in from the current frame and processes it through two layers of multilayer perceptrons (MLPs). Since objects detected in the previous frame may disappear in the current frame, a zero-padded empty vector is concatenated with Q-in to represent the query for disappeared objects. Finally, a dot product operation is used to associate the query features updated in the previous frame with the refined target features of the current frame. The output of the dot product operation is an N * (M + ... 1) A matrix, where N represents the number of objects detected in the previous frame and M represents the number of object queries in the current frame; higher association scores in the matrix indicate a greater likelihood of association; during the training phase, the association process is treated as a classification task, and the cross-entropy loss function is used to calculate the loss between the estimated results of the object association module and the true results; finally, during the evaluation phase, a greedy search-based method is applied after object association to prevent duplicate associations. The data association and matching module based on LSTM and Transformer also includes a query enhancement module based on temporal information. This module is used to model the motion trend of objects and, combined with a query initialization module with prior information, can accurately predict and track the motion trajectory of objects by modeling the perceptual information of consecutive frames, thereby enhancing detection accuracy through tracking tasks. The specific implementation process is as follows: In the query enhancement module based on temporal information, the query of the previous frame is... And querying the current frame The query from the previous frame is passed to the cross-attention mechanism. As a query in this attention mechanism, features from the previous frame are aggregated into the corresponding query of the current frame. In this way, historical frame information can be aggregated into the current frame, and for newly generated... This will be directly used as the query input for the cross-modal attention mechanism in the detection phase.

2. The target detection and tracking system based on multi-source heterogeneous sensor information fusion according to claim 1, characterized in that, The multimodal feature extraction and BEV feature generation module: In the camera image processing branch, the image feature extraction network VoVNet is used to extract image features. Its spatial dimension is shown in formula (1), and its generation method is shown in formula (2). Where C, H, and W represent the feature size, image height, and image width, respectively. Indicates camera image input, This represents the extracted image features.

3. The target detection and tracking system based on multi-source heterogeneous sensor information fusion according to claim 1, characterized in that, The lidar-guided camera view conversion module: based on image features Generate BEV features for the image, specifically: BEV features from the previous frame camera BEV characteristics of the current frame lidar Fusion into query features and image features As the key and value in the attention mechanism, deformable cross-attention is performed with image features to update the camera BEV features. This process can be described as follows: in, The spatial prior, generated from the current frame's LiDAR BEV features and the previous frame's camera BEV features, serves as the query in the attention mechanism. It's a connection operation. It is the BEV feature of the final generated camera image. It is a deformable attention operation. It is the image feature of the current frame.

4. The target detection and tracking system based on multi-source heterogeneous sensor information fusion according to claim 1, characterized in that, The multimodal feature extraction and BEV feature generation module, in the lidar point cloud processing branch, firstly performs voxelization on the input lidar point cloud, using the trained VoxelNet network to convert the original lidar point cloud into a unified mesh; simultaneously, to improve computational efficiency, a three-dimensional sparse convolutional SECOND network is used to encode the features in the voxel space, forming a feature representation in the BEV space, with the following spatial dimensions: This process can be represented as: in, , These represent the voxelization and 3D sparse convolution operations of the lidar point cloud, respectively. This represents the BEV features of the generated LiDAR point cloud. This represents point cloud input.

5. A target detection and tracking system based on multi-source heterogeneous sensor information fusion according to claim 1, characterized in that, The multimodal feature extraction and BEV feature generation module, in the millimeter-wave radar point cloud processing branch, firstly employs columnar processing as a preprocessing step, transforming the original millimeter-wave radar point cloud data into a more standardized and structured representation; secondly, it introduces a network architecture based on a neighborhood attention mechanism as the backbone network to extract the BEV point cloud features of the millimeter-wave radar. The spatial dimension of the BEV point cloud features of the millimeter-wave radar is: The above process can be represented as: in, Indicates a cascading operation. This represents point cloud input. This indicates the point cloud columnar processing of millimeter-wave radar. The attention mechanism representing neighborhood attention.

6. The target detection and tracking system based on multi-source heterogeneous sensor information fusion according to claim 1, characterized in that, The processing procedure of the query initialization module with prior information can be represented as follows: in, Indicates the query code. Indicates position code, Indicates content encoding. This indicates enhanced time-series information. Indicates a feedforward network. Indicates a connection operation. Represents the BEV features of a camera image. This represents the BEV feature of the lidar point cloud. This represents the BEV characteristics of millimeter-wave radar point clouds.

7. The target detection and tracking system based on multi-source heterogeneous sensor information fusion according to claim 1, characterized in that, The feature fusion module based on the cross-modal attention mechanism: First, a deformable attention mechanism is used to assign weights to features from different modalities, and then the features are fused by summing the weights. （12） In the above formula, first, F j The embedded learnable query encoding, position encoding, and content encoding are then used to... i Convert to query z qi and reference point p qi and F j Treated as keys and values, F i and F j These represent point cloud features from image, LiDAR, and millimeter-wave radar, respectively. `m` is used to index the attention head, `k` is used to index the sampling key, and `∆P`... mqk Indicates the sampling offset, A mqk Indicates attention weight; This formula updates F through a deformable cross-attention mechanism. j By exchanging F i and F j The three features represented are then updated through six different feasible variable cross-attention mechanisms to update image features, lidar point cloud features, and millimeter-wave radar point cloud features. Then, two feedforward networks are used to predict the 3D bounding boxes and category information of the detected targets based on the updated query. This prediction process is described as follows: in, and These represent feedforward networks for regression and classification tasks, respectively, and Query is the updated object query. Finally, binary matching is used to assign a one-to-one relationship between predicted and true values, focus loss is used for classification, and L1 loss is used for 3D bounding box regression. in, Represents the total loss function. It is a classification loss function that calculates the predicted classification information. With real classified information The losses between This represents the regression loss function, used to calculate the predicted bounding box information. Compared with the actual bounding box information The losses between and These represent the weight hyperparameters of the classification loss term and the weight hyperparameters of the regression loss term, respectively.

8. A target detection and tracking method based on multi-source heterogeneous sensor information fusion, characterized in that, The target detection and tracking system according to any one of claims 1-7 is deployed to a vehicle to achieve real-time target detection and tracking.

Images