Automatic driving roadside collaborative awareness method based on variability implicit feature alignment
By adopting a method based on deformable implicit feature alignment in the roadside collaborative perception system of autonomous driving, and using deformable convolution and attention mechanisms, the problem of feature alignment in heterogeneous sensor environment is solved, achieving higher perception accuracy and reliability.
Patent Information
- Application Number
- CN202411827192.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-04-04
AI Technical Summary
In heterogeneous sensor environments, it is difficult for the prior art to achieve accurate and reliable feature alignment, resulting in incomplete and inaccurate perception, affecting the safety and efficiency of autonomous vehicles.
Using a method based on deformable implicit feature alignment, through deformable convolution and attention mechanism, we can dynamically adapt the characteristics of different sensor types to achieve flexible alignment and fusion of features.
It realizes more accurate feature alignment and fusion in a heterogeneous sensor environment, improves the accuracy and reliability of roadside collaborative perception on autonomous driving, and significantly improves perception accuracy.
Smart Images
Figure CN119760502A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to vehicle-road cooperative technology and computer vision processing technology, and specifically to an autonomous driving roadside cooperative perception method based on variability implicit feature alignment. Background Art
[0002] The rapid development of autonomous driving technology has stimulated great interest in improving vehicle perception capabilities. A key aspect of this evolution is ensuring that vehicles can accurately and effectively perceive their surroundings. Traditional vehicle perception systems rely heavily on onboard sensors, which, while effective, have inherent limitations in range and field of view. These limitations can result in blind spots and incomplete situational awareness, which can affect the safety and efficiency of autonomous vehicles. To address these challenges, the concept of roadside collaborative perception has emerged as a promising solution.
[0003] Roadside cooperative perception uses infrastructure installed on the road, such as cameras, lidar, and radar sensors, to enhance the perception capabilities of individual vehicles. By sharing perception information between vehicles and roadside units (RSUs), roadside cooperative perception systems can provide a comprehensive understanding of the traffic environment that a single onboard sensor could not. This cooperative approach is particularly beneficial in complex urban environments, where obstacles such as buildings and other vehicles can block the line of sight of onboard sensors.
[0004] Despite the great potential of cooperative roadside sensing, there is a notable lack of research that develops algorithms specifically for this framework. Most existing studies on cooperative roadside sensing draw insights from vehicle-to-vehicle (V2V) or vehicle-to-road (V2X) collaboration and focus mainly on homogeneous traffic environments. These studies usually assume that all agents, including vehicles and roadside units (RSUs), are equipped with similar sensor types. This assumption, while simplifying the problem, fails to accurately reflect the complexity of contemporary traffic systems. Modern traffic systems are characterized by a diverse range of sensor types and configurations, which highlights the need for more nuanced approaches. Therefore, there is a large research gap in the context of cooperative roadside sensing, and focused research is needed to address the challenges posed by heterogeneous sensor environments.
[0005] The main challenge of roadside cooperative perception systems with heterogeneous sensors lies in the alignment of features extracted from different types of sensors. Each type of sensor, whether camera, lidar, or radar, captures different aspects of the environment, resulting in different representations of the same scene. For example, cameras provide high-resolution color images, while lidar captures accurate 3D point clouds and radar provides reliable distance measurements in various weather conditions. Aligning these disparate features into a cohesive representation is a non-trivial task that is essential for accurate and reliable perception. Traditional feature alignment methods usually assume that the sensor setup is homogeneous and are not suitable for handling the variability introduced by different types of sensors. In addition, simply concatenating or averaging features from different sensors can lead to suboptimal results because these methods do not take into account the inherent differences in sensor data. Advanced techniques are urgently needed to dynamically adapt to the characteristics of each sensor type and achieve robust feature alignment.
[0006] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention
[0007] The purpose of the present invention is to provide an autonomous driving roadside collaborative perception method based on deformable implicit feature alignment, so as to achieve accurate and reliable perception in a heterogeneous sensor environment.
[0008] In order to achieve the above object, the present invention proposes an autonomous driving roadside collaborative perception method based on deformable implicit feature alignment, comprising:
[0009] Step 1: LiDAR captures point cloud data;
[0010] Step 2: The captured point cloud data first inputs the lidar data into the backbone network for feature extraction;
[0011] Step 3: Transmit the intermediate features to various infrastructures;
[0012] Step 4: Each agent sends the aggregated intermediate features to DIFA for feature alignment;
[0013] Step 5: Send the aligned features to AFF for feature fusion;
[0014] Step 6: Conduct experiments on the dataset.
[0015] Preferably, in the above technical solution, step 1 of obtaining laser radar captured data is that the laser radar device performs a field scan to obtain the distance information of the target object.
[0016] Preferably, in the above technical solution, step 2 requires preprocessing the data, converting the point cloud data to the vehicle coordinate system, and determining the area of interest, and then calculating the smoothness of each point, marking the occluded points and extracting features, and finally inputting the point cloud data after preprocessing and feature extraction into the backbone network.
[0017] Preferably, in the above technical solution, step 3 needs to compress the infrastructure features to reduce the size of the transmission features and improve the transmission efficiency.
[0018] Preferably, in the above technical solution, the spatial position information perceived by each agent in step 4 is compressed from 3D information into a two-dimensional bird's-eye view, using the feature map F∈R C×H×W express.
[0019] Preferably, in the above technical solution, the improvement of the feature alignment method in step 4 includes:
[0020] (1) Using deformable convolution as the feature alignment function of the present invention can not only flexibly align spatial differences, but also effectively cope with the situation of multi-scale targets. The mathematical expression of deformable convolution can be expressed as:
[0021]
[0022] where R is augmented by an offset {Δp n |n=1,…,N}, where N=|R|, the deformable convolution adaptively learns the offset Δpn of different agent feature maps.
[0023] Preferably, in the above technical solution, the method for performing feature alignment in step 4 can be expressed by a mathematical expression as follows:
[0024] [F′1,F′2]=f align (f offset ([F1,F2]))
[0025] where [F1, F2] represents the concatenation of the agent’s feature maps F1 and F2, and f offset and f align They represent the functions of learning pixel offsets and aligning features with the learned offsets, and [F1′, F′2] are the aligned features.
[0026] Preferably, in the above technical solution, the DIFA-based feature alignment method in step 4 includes:
[0027] (1) First, without changing the resolution, a 1x1 convolution operation is used to perform linear transformation and feature integration on the feature map;
[0028] (2) Use deformable convolution to implement foffset and f align function;
[0029] (3) Use an activation function with the same kernel size.
[0030] Preferably, in the above technical solution, the attention mechanism of the AFF module in step 5 operates as follows:
[0031] (1) Query, key, and value calculation. For each input feature map, the present invention calculates query (Q), key (K), and value (V) matrices by learning linear transformations. These matrices capture different aspects of feature interactions. The present invention uses a scaled dot product attention mechanism, the core idea of which is to obtain attention weights by calculating and scaling the dot product between Q and key k, and then apply these weights to value V to obtain fused contextual information.
[0032] (2) Attention weight calculation. First, the input features are rearranged to fit the attention calculation, the score matrix is calculated by the dot product of the query matrix and the key matrix, and the attention weight matrix is obtained by soft-max normalization.
[0033] (3) Multi-head attention. In order to capture a richer set of interactions, the present invention uses multiple attention heads. Each head calculates its own set of attention weights and produces a sum of weighted values. The outputs of all heads are connected and linearly transformed to produce the final attention weighted feature map. Finally, the context information is obtained by weighted summation. Mathematically, AFF is expressed as:
[0034]
[0035] Where Q in , K in , V in represents the original matrix without linear changes to preserve as much original feature information as possible. h is the current number of positive faces, and M is the total number of positive faces. For each head, i represents the feature index of the current agent, and j represents the feature index of all agents.
[0036] Preferably, in the above technical solution, step 6 includes conducting experiments on the dataset in two representative scenes (i.e., intersections and corridors) to capture different traffic densities and configurations; and calculating the cross-link (IOU) for target detection to measure the overlap between the predicted and ground truth bounding boxes.
[0037] Compared with the prior art, the present invention has the following beneficial effects:
[0038] (1) The present invention proposes a roadside collaborative perception method for autonomous driving based on deformable implicit feature alignment. This method is based on intermediate fusion and combines deformable convolution to adapt to the unique characteristics of each sensor type, thereby achieving more accurate feature alignment.
[0039] (2) The present invention proposes an autonomous driving roadside collaborative perception method based on deformable implicit feature alignment, and adds an attention mechanism on the original basis. In sensor fusion, the attention mechanism can dynamically adjust the importance of different features according to their relevance to the perception task, thereby improving the overall perception accuracy.
[0040] (3) A large number of experiments were conducted on the DAIR-RCooper dataset. The results show that the framework proposed in the present invention exhibits strong performance in both intersection scenarios with heterogeneous data and corridor scenarios with homogeneous data, and shows superior performance under all IOU thresholds. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 is a schematic diagram of the roadside collaborative perception framework proposed by the present invention;
[0042] Figure 2 is a schematic diagram of a deformable implicit feature alignment module;
[0043] Figure 3 is a schematic diagram of the attention feature fusion module;
[0044] Figure 4 It is a schematic diagram of the process of the present invention;
[0045] Figure 5 It is a qualitative visual comparison diagram of three typical scenarios. DETAILED DESCRIPTION
[0046] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.
[0047] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.
[0048] like Figures 1 to 4 As shown, according to a specific embodiment of the present invention, a method for promoting effective feature alignment in a roadside cooperative perception system with heterogeneous sensors includes the following steps:
[0049] Step 1: LiDAR captures point cloud data
[0050] The laser radar device performs field scanning to obtain the distance information of the target object.
[0051] Step 2: The captured point cloud data first inputs the lidar data into the backbone network for feature extraction
[0052] The data needs to be preprocessed first, the point cloud data needs to be converted to the vehicle coordinate system, and the area of interest needs to be determined. Then the smoothness of each point needs to be calculated, the occluded points need to be marked, and the features need to be extracted. Finally, the point cloud data after preprocessing and feature extraction needs to be input into the backbone network.
[0053] Step 3: Transfer intermediate features to various infrastructures
[0054] Infrastructure features need to be compressed to reduce the size of transmission features and improve transmission efficiency.
[0055] Step 4: Each agent sends the aggregated intermediate features to DIFA for feature alignment
[0056] Step 4: Improvements to the feature alignment method include:
[0057] (1) Using deformable convolution as the feature alignment function of the present invention can not only flexibly align spatial differences, but also effectively cope with the situation of multi-scale targets. The mathematical expression of deformable convolution can be expressed as:
[0058]
[0059] where R is augmented by an offset {Δp n |n=1,…,N}, where N=|R|, the deformable convolution adaptively learns the offset Δpn of different agent feature maps.
[0060] The method for feature alignment can be expressed mathematically as follows:
[0061] [F1′,F2′]=f align (f offset ([F1,F2]))
[0062] where [F1, F2] represents the concatenation of the agent’s feature maps F1 and F2, and f offset and f align They represent the functions of learning pixel offsets and aligning features with the learned offsets, and [F1′, F2′] are the aligned features.
[0063] The feature alignment method of DIFA in step 4 includes:
[0064] (1) First, without changing the resolution, a 1x1 convolution operation is used to perform linear transformation and feature integration on the feature map;
[0065] (2) Use deformable convolution to implement f offset and f align function;
[0066] (3) Use an activation function with the same kernel size.
[0067] Step 5: Send the aligned features to AFF for feature fusion
[0068] The attention mechanism of the AFF module operates as follows:
[0069] (1) Query, key, and value calculation. For each input feature map, the present invention calculates query (Q), key (K), and value (V) matrices by learning linear transformations. These matrices capture different aspects of feature interactions. The present invention uses a scaled dot product attention mechanism, the core idea of which is to obtain attention weights by calculating and scaling the dot product between Q and key k, and then apply these weights to value V to obtain fused contextual information.
[0070] (2) Attention weight calculation. First, the input features are rearranged to fit the attention calculation, the score matrix is calculated by the dot product of the query matrix and the key matrix, and the attention weight matrix is obtained by soft-max normalization.
[0071] (3) Multi-head attention. In order to capture a richer set of interactions, the present invention uses multiple attention heads. Each head calculates its own set of attention weights and produces a sum of weighted values. The outputs of all heads are connected and linearly transformed to produce the final attention weighted feature map. Finally, the context information is obtained by weighted summation. Mathematically, AFF is expressed as:
[0072]
[0073] Where Q in , K in , V in represents the original matrix without linear changes to preserve as much original feature information as possible. h is the current number of positive faces, and M is the total number of positive faces. For each head, i represents the feature index of the current agent, and j represents the feature index of all agents.
[0074] Step 6: Experiment on the dataset
[0075] The DAIR-RCooper dataset released by Tsinghua University is a large real-world dataset designed for roadside collaborative perception research. It includes different traffic scenarios, and data is collected from multiple sensor types, including cameras and lidar installed on the roadside. The dataset includes two representative scenarios (i.e., intersections and corridors), capturing different traffic densities and configurations. The training process includes optimizing the network using the Adam optimizer. The initial learning rate is set to 2×10 -3 , and tuned according to the cosine annealing strategy, using a batch size of 16 and training the model for 50 epochs to ensure convergence.
[0076] We compute the Interconnection Over Union (IOU) for object detection to measure the overlap between the predicted and ground truth bounding boxes as an evaluation metric. We evaluate the performance at IOU thresholds of 0.3, 0.5, and 0.7 to obtain different accuracy levels. We compute the mean average precision (mAP) to evaluate the overall detection performance for different object categories. We compare our method with several state-of-the-art methods for heterogeneous sensor fusion and collaborative perception.
[0077] Experimental results show that the proposed method exhibits superior performance at all IOU thresholds. Specifically, at an IOU threshold of 0.3, the proposed model achieves 55.0% and 69.4% mAP in intersection and corridor scenarios, respectively, while the mAP of the SOTA method is 67.6% and 53.5%, respectively. At higher thresholds of 0.5 and 0.7 IOUs, the proposed model is still in the lead, with an average mAP improvement of more than 2% in intersection and corridor scenarios, significantly outperforming the baseline method.
[0078] To further understand the contribution of each component, the present invention conducted an ablation study by selectively disabling the DIFA and AFF modules.
[0079] Experimental results show that removing the DIFA module leads to a significant drop in IOU and mAP scores, highlighting the importance of deformable convolutions for feature alignment. Disabling the AFF module also leads to a drop in performance, indicating that the attention mechanism plays a crucial role in capturing spatial relationships and enhancing feature fusion. The combination of DIFA and AFF modules produces the best performance, demonstrating the synergistic effect of these components in improving perceptual accuracy.
[0080] Next, we evaluate the robustness of the model under common damage scenarios. Common damage means that two separate damage functions are applied to the point cloud data captured by the proxy to simulate real-world strong sunlight weather and sensor damage, such as Figure 5 shown.
[0081] Scenario 1 represents a corridor scenario where the agents are equipped with the same type of sensors and each method performs well. Scenario 2 represents a simple intersection scenario. There are data occlusion and data heterogeneity problems in this scenario. The method of the present invention can accurately detect occluded objects and pedestrians, but the other methods cannot locate the position of the 3D box well. Scenario 3 represents a complex intersection scenario. The huge traffic flow brings greater challenges to the data heterogeneity problem. The method of the present invention still maintains the best detection effect, while other methods even miss or misjudge the 3D box.
[0082] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.
Claims
1. A roadside collaborative perception method for autonomous driving based on deformable implicit feature alignment, characterized in that: A deformable implicit feature alignment module and an attention feature fusion module are proposed. The steps of the feature alignment method include: Step 1: LiDAR captures data; Step 2: The captured point cloud data first inputs the lidar data into the backbone network for feature extraction; Step 3: Transmit the intermediate features to various infrastructures; Step 4: Each agent sends the aggregated intermediate features to DIFA for feature alignment; Step 5: Send the aligned features to AFF for feature fusion; Step 6: Conduct experiments on the dataset.
2. The method for roadside collaborative perception of autonomous driving based on deformable implicit feature alignment according to claim 1, characterized in that: Step 1 is to obtain the laser radar captured data, and the laser radar device performs field scanning to obtain the distance information of the target object.
3. The method for roadside collaborative perception of autonomous driving based on deformable implicit feature alignment according to claim 1, characterized in that: Step 2 requires preprocessing the data, converting the point cloud data to the vehicle coordinate system, and determining the area of interest. It then calculates the smoothness of each point, marks occluded points, and extracts features. Finally, the preprocessed and feature-extracted point cloud data is input into the backbone network.
4. The method for roadside collaborative perception of autonomous driving based on deformable implicit feature alignment according to claim 1, characterized in that: Step 3 requires compressing the infrastructure features to reduce the size of the transmission features and improve transmission efficiency.
5. The method for roadside collaborative perception of autonomous driving based on deformable implicit feature alignment according to claim 1, characterized in that: In step 4, the spatial position information perceived by each agent is compressed from 3D information to a 2D bird's-eye view, using the feature map F∈R C×H×W express.
6. The method for roadside collaborative perception of autonomous driving based on deformable implicit feature alignment according to claim 1, characterized in that: Step 4 improves the feature alignment method by: (1) Using deformable convolution as the feature alignment function of the present invention can not only flexibly align spatial differences, but also effectively cope with the situation of multi-scale targets. The mathematical expression of deformable convolution can be expressed as: where R is augmented by an offset {Δp n |n=1,…,N}, where N=|R|, the deformable convolution adaptively learns the offset Δpn of different agent feature maps.
7. The method for roadside collaborative perception of autonomous driving based on deformable implicit feature alignment according to claim 1, characterized in that: The method for feature alignment in step 4 can be expressed mathematically as: [F1′,F2′]=f align (f offset ([F1,F2])) where [F1, F2] represents the concatenation of the agent’s feature maps F1 and F2, and f offset and f align They represent the functions of learning pixel offsets and aligning features with the learned offsets, and [F1′, F2′] are the aligned features.
8. The method for roadside collaborative perception of autonomous driving based on deformable implicit feature alignment according to claim 1, characterized in that: The DIFA-based feature alignment method in step 4 includes: (1) First, without changing the resolution, a 1x1 convolution operation is used to perform linear transformation and feature integration on the feature map; (2) Use deformable convolution to implement f offset and f align function; (3) Use activation functions with the same kernel size.
9. The method for roadside collaborative perception of autonomous driving based on deformable implicit feature alignment according to claim 1, characterized in that: The attention mechanism of the AFF module in step 5 operates as follows: (1) Query, key, and value calculation: For each input feature map, query (Q), key (K), and value (V) matrices are calculated by learning linear transformations. These matrices capture different aspects of feature interactions. The present invention uses a scaled dot product attention mechanism. The core idea is to obtain attention weights by calculating and scaling the dot product between Q and key k, and then apply these weights to the value V to obtain fused context information. (2) Attention weight calculation: First, the input features are rearranged to fit the attention calculation, the score matrix is calculated by the dot product of the query matrix and the key matrix, and the soft-max normalization is performed to obtain the attention weight matrix; (3) Multi-head attention: In order to capture a richer set of interactions, the present invention uses multiple attention heads. Each head calculates its own set of attention weights and produces a sum of weighted values. The outputs of all heads are connected and linearly transformed to produce the final attention weighted feature map. Finally, the context information is obtained by weighted summation. Mathematically, AFF is expressed as: Where Q in , K in , V in represents the original matrix that has not undergone linear changes to retain as much original feature information as possible, h is the current number of positive faces, M is the total number of positive faces, for each head, i represents the feature index of the current agent, and j represents the feature index of all agents.
10. The method for roadside collaborative perception of autonomous driving based on deformable implicit feature alignment according to claim 1, characterized in that: Step 6 conducts experiments on the dataset including two representative scenarios (i.e., intersection and corridor) that capture different traffic densities and configurations; and computes the intersection of unions (IOU) for object detection to measure the overlap between predicted and ground truth bounding boxes.