Multi-vehicle cooperative sensing method for mode deficiency in automatic driving scene

Abstract

The invention provides a multi-vehicle cooperative sensing method for modal deficiency in an automatic driving scene. The method can keep stable 3D target detection performance when the modal deficiency occurs. According to the method, the difference of BEV features of the laser radar and the camera in formats and dimensions is eliminated by adopting a multi-modal feature alignment strategy, and the fusion effect is improved; in order to enhance the adaptability of the model during mode missing, a cross-mode deformable attention layer is adopted to enable a linear layer with a specific mode to better adapt to data changes, and a long-distance dependency relationship between self-attention capture features is utilized. And finally, screening high-reputation neighbors by evaluating vehicle states and reputation values, and adopting a multi-attention multi-vehicle collaborative perception strategy to balance perception precision and reasoning efficiency. A large number of simulation and real vehicle experiment results show that the AP (at) 0.5 of the method is improved by 7.73% and 6.78% respectively when the laser radar or the camera is missing, and the method is obviously superior to an existing reference method.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and specifically is a multi-vehicle collaborative perception method for modal loss in an autonomous driving scenario. Background Art

[0002] As an innovative technology to improve traffic safety and efficiency, autonomous driving aims to minimize traffic accidents. This technology uses advanced perception, communication, computing, and control methods to achieve real-time and continuous autonomous control of autonomous vehicles. With the rapid development of artificial intelligence, the environmental perception capabilities of autonomous vehicles have been significantly improved. Generally, autonomous driving systems rely on on-board sensors (such as lidar, cameras, millimeter-wave radar, etc.) to continuously scan and monitor the surrounding environment to understand and perceive the environmental state [1]. However, in actual road environments, it is difficult for a single vehicle to obtain "perfect" perception data, and information is often missing due to reasons such as obstructed vision (as shown in Figure 1(a)). In addition, the data collected by different types of sensors are heterogeneous. For example, the camera collects two-dimensional RGB images that lack depth information, while the lidar generates sparse three-dimensional point clouds (as shown in Figure 1(b)). The diversity of this multi-source data significantly increases the difficulty of processing. How to improve the multi-source data fusion capability of a single vehicle, especially in severe weather conditions, is still an important challenge facing autonomous driving. Fortunately, there is a close interactive relationship between autonomous vehicles and surrounding traffic participants. By sharing perception data, multi-vehicle collaborative perception can effectively overcome the perception limitations of a single vehicle in the temporal and spatial dimensions, enabling vehicles to more comprehensively perceive road conditions and the dynamic behaviors of surrounding traffic participants, thereby providing more efficient and accurate support for autonomous driving decisions[2].

[0003] According to the different types of shared data, the existing multi-vehicle collaborative perception methods are mainly divided into camera-based [3][4][5], lidar-based [6][7][8] and multimodal fusion-based methods [9]

[10]

[11] . Among them, the multimodal fusion method has become the most promising perception scheme by combining the advantages of lidar and camera. However, most studies consider an ideal multimodal data input condition to seek the best 3D object detection performance

[12]

[13] , while there are relatively few studies on 3D object recognition in the case of missing modal data. For example, in the bad weather conditions shown in Figure 1(c), lidar is prone to misjudge rain and fog as obstacles, which may cause traffic accidents; while in high-speed driving or low-light environments, the images captured by the camera are easily blurred or darkened, weakening the multi-vehicle collaborative perception capability. Therefore, multi-vehicle multi-source data fusion in response to missing modal data poses new challenges to the reliability and robustness of autonomous driving systems.

[0004] (1) Accurate alignment of multimodal features. Since the data collected by lidar and camera have significant differences in format and dimension, it is difficult for the detection model to accurately align these heterogeneous data at the input stage or in the feature space. Eliminating the data differences relies on a unified representation suitable for multimodal feature fusion, in which depth information is the key to data alignment. The two-dimensional images collected by the camera lack depth information, while the lidar provides sparse three-dimensional point clouds, so it is necessary to align the two types of data in the depth dimension. Yin et al.

[14] projected the three-dimensional point cloud into the two-dimensional RGB camera coordinate system to help the camera obtain depth information, and performed 3D target detection in the surrounding environment based on the aligned data. However, the projection method is prone to information loss when the target is occluded and it is difficult to capture the depth information of distant objects. You et al.

[15] converted the estimated RGB image depth map into a pseudo lidar point cloud to achieve lidar-based 3D target detection. Reference

[16] proposed using a neural network to predict the depth information of the image, so as to construct an aligned BEV feature after generating a frustum point cloud from multi-view images. Although these methods reduce the dependence on LiDAR point cloud data to a certain extent, they are easily affected by noise and environmental changes, resulting in unstable depth prediction. To this end, the BEVFusion method

[12] uniformly projects multimodal features into the BEV representation space to efficiently fuse multimodal features. However, how to accurately align multimodal features in the BEV space remains a key problem.

[0005] (2) Robustness of multimodal data fusion. Complete multimodal perception data is the basis for realizing multimodal feature fusion. Reference

[17] proposed a query indexing method to realize multimodal data fusion by mutual query of lidar and camera features. However, when the lidar or camera fails or in harsh environments, this method significantly reduces the fusion effect due to its reliance on complete data input. References

[12]

[18] use a feature concatenation method to input multimodal features into the BEV encoder in sequence to correct the spatial misalignment of lidar and camera BEV features. However, when modal data is missing, this method is difficult to adaptively adjust the spatial misalignment, which has a negative impact on the accuracy of 3D target detection. Bai et al.

[19] used the attention mechanism to enhance the contribution of high-quality single-modal data in multimodal fusion, so that the vehicle can focus more on 3D targets in the environment. However, the traditional attention mechanism is difficult to effectively weight and aggregate missing modal data and complete modal data, which weakens its effect in multimodal fusion. In summary, modality loss makes the existing multimodal fusion strategy ineffective and even hinders the realization of 3D target detection. Therefore, how to ensure the robustness of data fusion under incomplete multimodal perception data needs further exploration.

[0006] (3) Efficient collaborative utilization of multimodal information. Early collaborative perception research tended to use methods such as concatenation, summation, or linear weighting to fuse multi-vehicle features

[20] . Although these methods are simple to operate and have fast inference speed, they ignore the potential correlation between multi-vehicle features, which limits the improvement of collaborative perception performance. Graph-based collaborative perception can propagate and aggregate the feature information of neighboring vehicles. Therefore, Wang et al. [7] first adopted a spatially adaptive graph neural network to model multi-vehicle collaborative relationships. Li et al.

[21] calculated the collaborative weight to deduce the correlation between vehicles in the collaborative graph to optimize information sharing and perception efficiency. Xu et al.

[22] used the self-attention mechanism to capture the multi-vehicle relationship in a specific area of ​​the feature map and enhanced the overall feature fusion effect by assigning precise attention weights to different areas. Cui et al.

[23] designed a self-attention network called Point Transformer to aggregate the point cloud features of multiple collaborative vehicles to enhance feature extraction capabilities. Liu et al.

[24] compressed the perception information of the central vehicle and broadcast it to the surrounding vehicles, and built a collaborative relationship by learning matching scores. These methods mainly rely on the fusion of single-modal data to achieve multi-vehicle collaborative target detection, but fail to fully exploit the rich environmental information provided by multi-modal fusion data. Therefore, how to efficiently utilize the multi-modal features of neighboring vehicles and improve the target detection accuracy of the center vehicle is still full of challenges. Summary of the invention

[0007] In view of the above problems, the present invention proposes a multi-vehicle collaborative perception method based on multimodal fusion to compensate for the impact of insufficient environmental perception information on 3D target detection performance.

[0008] The multi-vehicle collaborative perception method for modality loss in an autonomous driving scenario of the present invention is applied to an autonomous driving scenario; in an autonomous driving scenario, a vehicle continuously scans and monitors the surrounding environment through an on-board laser radar and camera to perceive and understand the state of the environment;

[0009] The steps of the multi-vehicle collaborative perception method include:

[0010] (i) First, the BEV features obtained by the vehicle's lidar and camera are fused to obtain a multi-modal fused BEV feature B;

[0011] (2) A multi-vehicle collaborative perception method based on a multi-head attention mechanism is used to select collaborative neighbor vehicles;

[0012] (iii) Use a 3D target detection head to identify multiple targets around the vehicle.

[0013] The main contributions of the present invention are as follows:

[0014] First, a multimodal feature alignment strategy combining projection and prediction. This strategy projects the depth information of the lidar into the RGB image of the camera to enhance the depth perception capability of the camera; then the aligned features are converted to the BEV space and fused with the BEV features of the lidar. This method effectively eliminates the differences in format and dimension between the lidar and camera data, making the aligned features more suitable for multimodal data fusion.

[0015] Second, the design of cross-modal deformable attention layers and self-attention layers. By using modality-specific linear layers instead of traditional unified linear layers, the model can better adapt to input data in different environments, especially when modal data is missing. At the same time, the self-attention layer is used to capture the long-range dependencies of multimodal features within the feature map, further improving the feature expression capability. This design ensures the effective fusion of multimodal data under various weather and traffic conditions (such as rainy days, foggy days, and high-speed driving), and enhances the system's robustness in 3D object detection in complex environments.

[0016] Third, a multi-vehicle collaborative perception strategy based on a multi-head attention mechanism. In the constructed collaborative network, the vehicle's status and demand information are used to evaluate the collaborative reputation value, helping the vehicle select neighbors with high reputation values ​​for collaboration. The multi-head attention mechanism is used to fuse the BEV features of multiple vehicles to balance the multimodal perception accuracy and model reasoning efficiency.

[0017] Experimental results on two autonomous driving collaborative perception datasets show that the proposed method improves the efficiency and robustness of 3D object detection through precise alignment and effective fusion of multimodal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1(a) to Figure 1(c) They represent three cases of vehicle perception in autonomous driving scenarios (i.e., field of view obstruction, data heterogeneity, and bad weather).

[0019] Figure 2(a) shows the multimodal fusion method.

[0020] Figure 2(b) shows the multi-vehicle collaborative perception framework based on multimodal fusion.

[0021] Figure 3 Represents a view frustum point cloud.

[0022] Figure 4 Represents a cross-modal deformable attention mechanism.

[0023] Figure 5 Represents the impact of the number of cooperative vehicles on the performance on the OPV2V dataset.

[0024] Figure 6(a) and Figure 6(b) respectively show the impact of different positioning errors on the performance.

[0025] Figure 7(a) and Figure 7(b) respectively show the impact of different heading errors on the performance.

[0026] Figure 8 Represents a visual comparison of 3D detection results on the OPV2V dataset, where the green box and red box represent the true box and the detection box, respectively. DETAILED DESCRIPTION

[0027] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0028] 1. Overview of the Program

[0029] Collaborative sensing alleviates the information loss problem of a single vehicle under occlusion and bad weather by sharing the complementary perception information of neighboring vehicles. However, existing methods mostly focus on a single modality (such as lidar or camera) and fail to fully utilize the advantages of multi-modality.

[0030] The present invention proposes a multi-vehicle collaborative perception method based on multimodal fusion, which can maintain robust 3D target detection performance when the modality is missing.

[0031] The method includes a multimodal feature alignment strategy to eliminate the differences in format and dimension between the BEV features of the lidar and camera and improve the fusion effect.

[0032] To enhance the model’s adaptability when modality is missing, a cross-modal deformable attention layer uses modality-specific linear layers to better adapt to data changes and exploits self-attention to capture long-range dependencies between features.

[0033] Finally, a multi-vehicle collaborative perception strategy with multi-head attention is proposed to balance perception accuracy and reasoning efficiency by evaluating vehicle status and reputation value to screen high-reputation neighbors.

[0034] A large number of simulation and real-car experimental results show that the proposed method improves AP@0.5 by 7.73% and 6.78% respectively when the lidar or camera is missing, significantly outperforming the existing baseline methods.

[0035] 2. Design

[0036] As shown in FIG. 2( a ) and FIG. 2( b ), the multi-vehicle cooperative perception architecture based on multimodal fusion of the present invention mainly includes four interdependent functional modules:

[0037] BEV feature extraction module: The dual-branch network of lidar and camera extracts BEV features of two modalities;

[0038] Multimodal fusion module: The cross-modal deformable attention module fuses the BEV features of the two modalities;

[0039] Multi-vehicle collaborative perception module: The collaborative perception module projects the fused BEV features into the coordinate system of neighboring vehicles to implement multi-vehicle feature fusion;

[0040] 3D Object Detection Head: The 3D detection head identifies multiple objects around the vehicle.

[0041] 2.1BEV feature extraction module

[0042] BEV feature extraction from LiDAR

[0043] The vehicle uses PointPillars

[25] to extract point cloud features from the data collected by the lidar. First, the point cloud data is divided into H×W grids according to the two-dimensional plane. The point cloud data falling into the same grid is regarded as being in one Pillar and is represented by the vector l. 1 =(x,y,z,r) represents a point in the cylinder, where (x,y,z) represents the position information and r represents the reflection intensity. Consider the geometric center l of Pillar 2 =(x c ,y c ,z c ) and center offset l 3 =(x p ,y p )=(xx c ,yy c ), the point cloud is encoded into a 9-dimensional vector When the number of points in the point cloud column is regularized to T through random sampling and zero padding, the dimension of the point cloud becomes H×W×T×9. Then, the vehicle uses the PointNet network to extract the intermediate features of the point cloud Then the BEV features of the lidar are output through maximum pooling Where N is the number of feature channels.

[0044] BEV feature extraction of camera

[0045] Although the camera can obtain rich color and texture information, the lack of depth information will lead to certain limitations in the 3D spatial description of the image. In particular, every point in the physical space cannot be captured by all sensors at the same time, resulting in incomplete information.

[0046] In the following two data perception situations, projection and prediction are used to obtain the depth information of the image for generating BEV features.

[0047] Case 1 When the lidar and camera simultaneously obtain information about each point in the physical space, the vehicle can directly map the lidar coordinates to the image pixel coordinates, thereby obtaining high-quality image depth information. This mapping process requires a series of coordinate transformations, including the transformation from the lidar coordinate system to the camera coordinate system, the transformation from the camera coordinate system to the image coordinate system, and the transformation from the image coordinate system to the pixel coordinate system. Specifically, the lidar coordinate system is regarded as the world coordinate system, where the world coordinates of any point are (X W ,Y W ,Z W ) T Assume that the transformation matrix E from the laser radar coordinate system to the camera coordinate system is the camera's external parameter matrix. The point in the camera coordinate system can be expressed as

[0048]

[0049] Where (X W ,Y W ,Z W ,1) T and (X C ,Y C ,Z C ,1) T All represent homogeneous coordinates. Let the focal length of the camera be f, and the point (X C ,Y C ,Z C ) T is transformed into the image coordinate system, i.e.

[0050]

[0051] Where (x,y,1) T represents homogeneous coordinates, and K represents the intrinsic parameter matrix of the camera. Each pixel of the camera corresponds to a photosensitive point, and the physical size width and height of the matrix centered on this point are dx and dy respectively. The present invention discretizes the image coordinate system and establishes a pixel coordinate system with the upper left corner as the coordinate origin. The point (x, y) T is converted to the pixel coordinate system, that is,

[0052]

[0053] Formula (3) gives the pixel coordinate system representation of the camera image, denoted as In summary, the distance Z from the pixel point (u, v) to the camera plane is C That is the image depth information d.

[0054] Case 2: When the image pixel acquired by the camera does not have the corresponding LiDAR projection depth, the vehicle still needs to obtain reliable depth information to extract BEV features. At this time, the LSS algorithm

[16] is used to implicitly estimate the image depth to compensate for the missing depth information. The LSS algorithm generates a discrete point set of D depths for each pixel (u, v) Corresponding to D 3D points This point set generates D planes at a distance from the camera plane, so the image is represented as a frustum point cloud (refer to Figure 3 ). Then, EfficientNet

[26] extracts the D-dimensional depth features and N-dimensional semantic features of each pixel in the image. The projected depth calculated in Case 1 is mapped to a discrete depth value and combined with the depth features predicted by the Softmax function. The outer product operation is then performed with the semantic features to obtain the probability distribution of the point (u, v, d) and the image feature point cloud. Finally, calculate the 3D coordinates of the pixel (u, v) in the vehicle coordinate system

[0055] (x,y,z) T =d·(K·E) -1 (u,v,1) T (4)

[0056] Drawing on the above PointPillar method, the image feature point cloud F (C) According to formula (4), they are assigned to different Pillars. The image feature point cloud belonging to the same Pillar is summed and pooled to output the BEV feature map

[0057]

[0058] This feature map represents the image features around the central vehicle from a bird's eye view.

[0059] 2.2 Multimodal Fusion

[0060] Traditional multimodal fusion directly inputs the spliced ​​or tandem lidar and camera BEV features into the 3D detection head in the channel dimension to retain the original information as much as possible. However, the heterogeneity of lidar and camera will cause spatial dislocation and produce rough fusion features. In addition, if the data collected by the camera or lidar is missing, the simple splicing or tandem method will make the feature fusion strategy invalid. The present invention designs a cross-modal deformable attention layer (Cross-Modal Attention) and a self-attention layer (Self Attention) to solve the robustness problem of feature fusion when the modality is missing.

[0061] The traditional deformable attention mechanism

[27] uses a unified linear layer to generate sampling offsets and attention weights. This makes the model unable to accept different environmental input variables, especially in scenarios where modality data is missing. To this end, the present invention replaces the unified linear layer with a modality-specific linear layer to adapt to the feature extraction and fusion of multiple input variables.

[0062] like Figure 4 As shown, the present invention uses sine and cosine functions as query features Generate Position Embedding And P e Add Q element by element to get a query feature containing spatial location information At the same time, the spliced ​​lidar and camera BEV features V = (B (C) ,B (L) ) as the value feature, where B (C) or B (L) Allow missing. Set a scaling factor r, the present invention generates H on the value feature V p ×W p A grid of reference points and the value of any reference point p is a linearly spaced two-dimensional coordinate {(0,0),…,(H p -1,W p -1)}, where H p =H / r,W p =W / r. In order to describe the relative position, these reference points are normalized to [-1, +1], where (-1, -1) represents the upper left corner and (+1, +1) represents the lower right corner. The deformable attention mechanism of the present invention divides the feature map into M groups in the channel dimension to form M attention heads in order to enhance the representation ability and learning diversity of the model. Each attention head only focuses on the S key sampling points around the reference point p and calculates their value features V. In order to determine the position of these key sampling points, the query vector Q′ is input into two linear layer modules in parallel. Q′ passes through the laser radar linear layer (Linear-L) and the camera linear layer (Linear-C) Generates a position offset relative to the reference point p

[0063]

[0064] Concat(·) represents the channel concatenation operation. (C) or B (L)When missing, Q′ only passes through the linear layer of the non-missing mode and does not perform the splicing operation. In the sampling offset module, the reference point p is added to Δp to obtain the sampling position of the value feature V. The coordinate value of p+Δp may not be an integer, so Equation (7) selects the four points closest to p on V (a x,l ,a y,l ), l=1,…,4 perform bilinear interpolation to calculate the eigenvalues ​​of S key sampling points

[0065]

[0066] And get the sampled value features Where g(a,b) = max(0,1-ab). The present invention uses query, key, value and position offset to calculate multi-head attention, where the output of the mth attention head is

[0067]

[0068] At the same time, the query vector Q′ passes through the camera linear layer and the lidar linear layer Get splicing features Similar to the linear layer above, when B (C) or B (L) When missing, Q′ only passes through the linear layer of the non-missing modality and does not perform concatenation. Then, the concatenated feature is input into the cross-modal deformable attention through the Softmax function. The multi-head attention of the sampling bias module also corresponds to the design of M attention heads and generates cross-modal deformable attention weights A. A can be used to scale features To obtain cross-modal query features

[0069]

[0070] in and represents a learnable projection matrix.

[0071] In order to establish the internal relationship between query features, two self-attention layers are added after the four cross-modal deformable attention layers in Figure 2, and the two attention layers adopt similar structures. The query feature and the value feature are both Q″, so when constructing the self-attention layer, there is no need to use a modality-specific linear layer but only a unimodal linear layer to generate Δp and A. Therefore, the value feature V in Equation (9) can be replaced by Q″ to calculate the self-attention

[0072]

[0073] As a result, the vehicle uses the cross-modal deformable attention layer and the self-attention layer to establish the relationship between multimodal features and obtain the final multimodal fusion features. This feature will participate in multi-vehicle collaborative perception as the vehicle's BEV feature.

[0074] 2.3 Multi-vehicle Collaborative Perception

[0075] In order to realize multi-vehicle collaborative perception, the present invention designs a topological map To build a multi-vehicle cooperative network, is the point set composed of vehicles, is the edge set connecting the vehicles, is the point set of neighboring vehicles. Multi-vehicle collaborative perception can be divided into two steps: (1) vehicle From its neighbor set The neighbor point set to be cooperated is selected according to the credibility value Its cardinality is J i ; (2) Vehicle i and its selected neighbors perform multi-modal feature fusion based on multi-head attention to complete 3D object detection with high quality. The multi-vehicle collaborative perception process is described in detail below.

[0076] Vehicle i uses a set of convolutions to extract the multimodal BEV features B i Compressed into query features with required information And broadcast to all neighbor vehicles. Neighbor j generates BEV feature B using a set of convolutions i,j Key information carried Combined with the query feature ε i Calculating the reputation of dot product attention

[0077]

[0078] in Represents the parameter matrix of dot product attention. Usually, query information is broadcast by wireless link and key information is calculated locally, so O<<U is set to save bandwidth. When all neighbors send back reputation values, vehicle i normalizes this set of reputation values ​​using the Softmax function, that is,

[0079]

[0080] And select the probability value greater than the preset threshold Neighbors as cooperative vehicles. Vehicle i sends a cooperation request to the selected neighbor vehicles, and they immediately reply and send feature maps after receiving the request information. Received cooperative feature map set After that, vehicle i and neighbor j perform feature map fusion based on multi-head attention. Suppose the expression of an attention head h is

[0081]

[0082] in and represents the parameter matrix of the linear layer, Represents the attention weight matrix that computes the dot product of the query vector and the key vector. The multi-head attention corresponding to the cooperative vehicle is

[0083] MultiHeadATT i,j =[head i,j,1 ,head i,j,2 …,head i.j,h ] (14)

[0084] In order to quantify the correlation between vehicle i and its cooperative neighbor j, the present invention generates the multi-head attention weight based on the normalized reputation value by equation (12) and equation (14):

[0085] W i,j =MultiHeadATT i,j γ i,j (15)

[0086] Based on this, the collaborative fusion features of vehicle i are generated

[0087]

[0088] Where FFN(·) is a feedforward neural network consisting of a set of linear layers and activation functions. Fusion feature B i ’ is input to the 3D detection head for target detection.

[0089] 2.4 3D Object Detection

[0090] Vehicle i inputs the collaborative fusion features into the 3D detection head to observe the objects around the vehicle body. The available 3D object detection models mainly include PointNet, SECOND, VoxelNet, AVOD, centerPoint, etc. The 3D detection head in this example uses the SSD (Single Shot Multibox Detector) model

[28] to predict the confidence of the 3D bounding box and the target category. SSD consists of two main components: the classification head and the regression head. The classification head predicts the confidence of the object category in each predicted box, and the regression head predicts the offset of each predicted box relative to the true box. The 3D detection head uses the target classification loss during model training. and regression loss To optimize its own parameters. Specifically, the present invention first gives a real frame vector Where (x, y, z) represents the target center, represents the size of the 3D box, and θ represents the rotation angle of the target. The Focal loss

[29] of formula (17) is used as the target classification loss to balance the foreground target and background area.

[0091]

[0092] Where μ represents the class probability, α 1 and α 2 is a hyperparameter. It defines the regression loss between the predicted value and the true value.

[0093]

[0094] Smooth L1 The loss

[30] can be used to supervise the size of the 3D object box

[0095]

[0096] Where θ′ represents the rotation angle of the prediction box. Therefore, the total loss of model training is

[0097]

[0098] where λ 1 and λ 2 They represent the classification loss coefficient and regression loss coefficient respectively.

[0099]

[0100] 3 Experiments and results analysis

[0101] In order to evaluate the detection performance of the proposed method in different traffic environments, two authoritative datasets, OPV2V

[22] and DAIRV2X

[31] , were selected for this experiment. The OPV2V dataset simulates V2V cooperative perception in the Carla simulator, where the number of vehicles ranges from 2 to 7. Each vehicle is equipped with a camera with 4 viewpoints and a lidar with a perception range of 40m×40m. The OPV2V dataset includes 6765 training samples, 1980 validation samples, and 2170 test samples. The DAIRV2X dataset is the first public real-world cooperative perception dataset, containing 100 real-world scenes and 18,000 samples. Each vehicle is equipped with a lidar and a front-view camera with a perception range of 201.6m×80m. The DAIRV2X dataset is divided into training, validation, and test sets in a ratio of 5:2:3.

[0102] The method proposed in the present invention is implemented under the PyTorch platform framework. The experimental environment of this experiment is built on a high-performance server. The server is equipped with an Intel Core i9-14900K processor, 64GB DDR5 5200MHz memory, 4TBPCIe 4.0 solid-state drive, ASUS PRIME Z790-P WIFID5 motherboard, and two Gigabyte RTX 4090 24GB WindForce graphics cards. For the OPV2V dataset, the initial learning rate and attenuation weight coefficient are set to 0.002 and 0.01, respectively. The camera image size of the four perspectives is 800×600. The laser radar point cloud is [-140m, 140m], [-40m, 40m] and [-3m, 1m] along the XYZ axis. For the DAIRV2X dataset, the initial learning rate and attenuation weight coefficient are set to 0.001 and 0.0001, respectively. The lidar point cloud is [-100m, 100m], [-40m, 40m] and [-3m, 1m] along the XYZ axis. The average precision (AP) with intersection over union (IoU) thresholds of 0.5 and 0.7 is used to evaluate the 3D object detection performance.

[0103] To facilitate quantitative and qualitative analysis and comparison, the eight benchmark methods selected in Table 1 are trained and evaluated in a unified code framework. Under the default noise setting, the positioning and heading errors of the cooperative vehicle are set to 0.2m and 0.2°, respectively. Two typical benchmark methods were selected for the experiment: No Fusion is a single-vehicle perception method that only uses local data; Late Fusion is a multi-vehicle collaborative perception method that fuses multiple predicted bounding boxes to achieve 3D target detection. In addition, the most representative When2com

[32] , DiscoNet

[21] , V2VNet[7], Where2comm

[33] , V2X-ViT[8], and CoBEVT[4] were selected as benchmark methods.

[0104] Table 1 9 target detection methods

[0105]

[0106] 3.1 Quantitative Experiments

[0107] Table 2 3D target detection accuracy under different modalities

[0108]

[0109] This group of experiments investigates the impact of vehicle collaboration and modality selection on performance under the default noise setting. Because the default noise setting simulates an ideal traffic scenario, Table 2 shows that the 3D target detection accuracy of the nine methods on the simulated OPV2V dataset is generally higher than that of the realistic DAIRV2X dataset, where L represents lidar and C represents camera. At the same time, compared with single-modal methods such as When2com, DiscoNet, and CoBEVT that only use lidar or camera, the two multi-modal methods Late Fusion and Proposed that fuse lidar and camera information show better detection results. Although No Fusion combines the features of the two modalities to anchor the target position, it is difficult to overcome the shortcomings of incomplete observations by relying solely on data collected by a single vehicle. Its AP@0.5 and AP@0.7 on the two datasets are not ideal. Multi-vehicle collaborative perception can help a single vehicle obtain multi-source environmental information, so the six target detection methods from When2com to CoBEVT have improved performance even if they only use single-modal collaboration. Among them, CoBEVT takes images from four perspectives and captures the target more accurately than five methods such as When2com and V2X-ViT that use lidar point clouds. Its AP@0.5 and AP@0.7 on two datasets have a maximum improvement of 11.55% and 11.67%. Proposed takes into account the advantages of multi-vehicle collaborative perception and multimodal fusion, and achieves the highest values ​​of AP@0.5 and AP@0.7 on two datasets. It is 5.53% to 4.42% higher than CoBEVT of multi-vehicle collaboration, 25% to 4.22% higher than No Fusion of multimodal fusion, and 11.4% to 6.59% higher than Late Fusion of multi-vehicle multimodal fusion. This shows that multimodal fusion can bring detection performance gains in both single-vehicle perception and multi-vehicle collaborative perception.

[0110] This set of experiments analyzes the impact of different numbers of cooperative vehicles on performance on the OPV2V dataset. Figure 5It shows that the AP@0.7 of the seven methods increases linearly with the increase in the number of cooperative vehicles. This is because the rich multi-source information provided by more cooperative vehicles can effectively reduce the false detection and missed detection caused by blind spots. However, Where2comm only focuses on the key areas of the lidar point cloud in order to reduce the communication cost, so its AP@0.7 is significantly lower than other methods. When the number of vehicles increases to 4 and 5, V2X-ViT and CoBEVT reduce the performance gain due to the use of single-modal cooperative perception. When the number of cooperative vehicles is 5, their AP@0.7 is 5.14% and 4.56% lower than the proposed method, respectively. This shows that the single-modal fusion strategy is difficult to eliminate the deviation of deep feature fusion when processing complex multi-source information due to the lack of assistance from other modal information. Proposed allows the BEV features of the lidar and camera point clouds to assist each other to enhance the mining of deep information, which increases AP@0.7 by 8.73% from 64.25% to 72.98%.

[0111] Figures 6(a) and 6(b) and Figures 7(a) and 7(b) verify the impact of different positioning and heading errors on performance. The positioning and heading errors are sampled at standard deviations of σ xyz = Gaussian distribution of [0,0.5m] and [0,1.0]. As shown in Figure 6(a), Figure 6(b), Figure 7(a) and Figure 7(b), although the AP@0.7 of the eight methods decreases with the increase of positioning and heading errors, the detection performance of Proposed on the two datasets is better than that of other methods. These results are due to the fact that Proposed not only uses a multi-head attention mechanism to select the most suitable collaborative features when fusing multi-modal features of multiple vehicles, but also filters out noise through the confidence of key information, thereby reducing the interference of positioning and attitude errors and enhancing the robustness of the model. In addition, the increase of the two errors aggravates the misalignment of BEV feature maps between different vehicles, which in turn causes the difficulty of collaborative perception. For example, when the positioning error on the DAIR-V2X dataset exceeds 0.2m or the heading error exceeds 0.2°, the AP@0.7 of V2VNet and Where2comm is even lower than that of No Fusion for single-vehicle perception. It is worth noting that the proposed method captures the key information and overall information perceived by different vehicles by constructing a multi-vehicle cooperative network, so the AP@0.7 on the OPV2V dataset is reduced by 9.46% and 10.35%, and the AP@0.7 on the DAIRV2X dataset is reduced by 9.63% and 5.18%.

[0112] 3.2 Ablation Experiment

[0113] Table 3 Ablation experiments on the number of cameras on the OPV2V dataset

[0114]

[0115] Table 3 shows the ablation experiment results for the number of cameras. Thanks to the effective combination of the LSS algorithm and the projection algorithm, when the number of cameras is reduced from 4 to 1, the AP@0.5 and AP@0.7 of Proposed decrease by 2.27% and 3.92% respectively. This gentle performance degradation shows that the LSS algorithm can make up for the information loss caused by the reduction of view angles, and can still obtain highly robust object detection results even when multimodal data is incomplete. Note that when the number of cameras is reduced from 3 to 2, the AP@0.5 and AP@0.7 of Proposed decrease less, 0.28% and 0.66% respectively; but when the number of cameras is reduced from 2 to 1, the AP@0.7 decreases more, 1.05%. This illustrates the limitations of single-view perception and detection. Nevertheless, the AP@0.5 and AP@0.7 of Proposed are higher than those of CoBEVT. Although CoBEVT designs a fusion axis attention to capture the local and global relationships between different cameras, it ignores the correlation between multiple perspectives and lacks the projection depth of the lidar, which leads to a significant decrease in collaborative perception performance when the number of cameras is small.

[0116] Table 4 Ablation experiment of modality loss on OPV2V dataset

[0117]

[0118] This group of experiments investigated the impact of modality loss on performance when the proposed method uses different feature fusion strategies. In Table 4, Max means taking the maximum value of each position in the feature map, Concat means concatenating feature maps, and DeformATT means using a fused feature map based on deformable attention. The DeformATT fusion strategy used by the proposed method more effectively captures the dependencies between multiple modalities through the deformable attention mechanism, and dynamically adjusts the weights of each modality to address the problem of decreased detection performance when the modality is missing. Table 4 shows that the AP@0.5 and AP@0.7 of DeformATT under the Lidar+Camera fusion strategy are 4.05% and 4.65% higher than Max, and 3.54% and 3.94% higher than Concat. In addition, when Lidar is missing, the three fusion strategies all show varying degrees of performance degradation, especially Max's AP@0.5 and AP@0.7 drop to 64.67% and 47.62%, respectively. In contrast, DeformATT can adaptively adjust the fusion process of missing and non-missing modalities, which makes AP@0.5 and AP@0.7 not drop significantly, which are 72.40% and 58.12% respectively. When the camera is missing, the detection performance of the three fusion strategies only decreases slightly, which shows that LiDAR scanning is easier to capture the depth information of the target than camera images, and accurate 3D target depth positioning will help improve detection accuracy. At this time, compared with the simple fusion strategies of Max and Concat, DeformATT dynamically adjusts the attention distribution to minimize the impact of incomplete camera information on target detection. The AP@0.5 of DeformATT is 6.78% and 4.69% higher than Max and Concat respectively. The above results show that the proposed DeformATT fusion strategy can ensure the adaptability and robustness of the detection model when the modality is missing.

[0119] Table 5 Ablation experiments of cross-modal deformable attention on OPV2V dataset

[0120]

[0121] As shown in Table 5, the ablation experiment results of cross-modal deformable attention show that different layer configurations have a significant impact on the detection performance. When the number of cross-modal layers increases from 4 to 6, the AP@0.5 of Proposed increases from 84.85% to 85.82%, and AP@0.7 increases from 69.71% to 70.35%. This shows that increasing the number of cross-modal layers can improve the detection accuracy of the model at different thresholds. However, the AP@0.5 and AP@0.7 corresponding to 8 layers of cross-modal layers drop to 85.14% and 70.35%, respectively, which may be due to performance saturation or information redundancy caused by too many layers. Adding 2 layers of self-attention to 4 layers of cross-modal to enhance feature extraction and fusion capabilities, the AP@0.5 of this method is 2.84% higher than that of only 4 layers of cross-modal layers. Similar to adding cross-modal layers, both cross-modal deformable attention and self-attention are increased to 4 layers. The deeper network structure leads to a complex optimization process, and the proposed AP@0.5 and AP@0.7 are reduced to 86.73% and 71.24% respectively. It is found that the reasonable configuration of cross-modal deformable attention layer and self-attention layer can further optimize the detection performance.

[0122] 3.3 Qualitative Experiments

[0123] In order to intuitively demonstrate the performance of different 3D object detection methods, Figure 8 Two typical autonomous driving scenarios, straight road and turning, were selected on the OPV2V dataset, and the visualization results under the default noise setting were given. It is obvious that No Fusion, which only uses its own perception information, has false detection in the second scenario. Where2comm improves communication efficiency while sacrificing some point cloud information of cooperative vehicles due to semantic compression. The reduced perception range causes Where2comm to miss detections in both scenarios. Compared with the first two methods, although the single-modal CoBEVT has fewer missed detections and false detections, the smaller intersection area between the true box and the detection box causes partial loss of the target. Proposed generates more predicted bounding boxes consistent with the true box, overcoming the above problems. Even if there is a blind spot in the view, Proposed can compensate for the occluded view through multimodal fusion and multi-vehicle collaboration, thereby minimizing the missed detection rate and false detection rate.

[0124] 3.4 Scalability Analysis

[0125] Based on the characteristics of the proposed multimodal multi-vehicle cooperative perception algorithm, a scalability analysis is performed to demonstrate its efficiency and robustness.

[0126] (1) Scalability of multimodal fusion. In autonomous driving scenarios, vehicles are usually equipped with multiple sensors, such as cameras and lidars, and the data they generate has multimodal characteristics. The proposed feature fusion method based on deformable attention can flexibly process data of different modalities and effectively fuse these heterogeneous data. With the increase in the types of sensors or the fluctuation in the quality of different sensor data, the proposed method can dynamically adjust the fusion strategy according to the contribution of each modality, thereby maintaining efficient and accurate perception capabilities. With the addition of more modalities, the multi-head attention mechanism of the proposed method can automatically adjust the weights of different modal information, effectively avoiding the excessive influence of a certain modality on the fusion result. The experimental results in Table 4 show that the proposed method can still maintain high-quality target detection performance when the number of multimodal sensors continues to increase, proving its scalability in processing multimodal data.

[0127] (2) Scalability of the multi-vehicle collaborative perception model. As the number of vehicles increases, the collaborative perception model needs to be able to process multimodal data from different vehicles while maintaining efficient feature fusion and target detection performance. The proposed multi-head attention-based feature fusion method has strong scalability and can support high-quality 3D target detection in a multi-vehicle environment. However, as the number of vehicles increases, the pressure on communication bandwidth and computing resources will inevitably increase. To meet this challenge, the proposed method can effectively reduce unnecessary data transmission and reduce the demand for computing and bandwidth resources by selecting neighbor vehicles with higher credibility values ​​for collaboration. Referring to the experimental results, especially in Figure 5 From the performance comparison in , we can see that the proposed method can still maintain a high target detection accuracy even in the case of high vehicle density, which proves its scalability in large-scale fleets.

[0128] 4 Conclusion

[0129] This paper proposes a multi-vehicle collaborative perception method for modality loss. In order to improve the multimodal data fusion effect of the vehicle itself, a multimodal feature alignment strategy combining projection and prediction is explored. The BEV features of the aligned lidar point cloud and camera RGB image are fused in the feature space through a cross-modal deformable attention layer and a self-attention layer, making full use of semantic and geometric information to alleviate the adverse effects of modality loss. A multi-vehicle collaborative perception strategy based on a multi-head attention mechanism is designed to improve the 3D target detection performance of the system by fusing the multimodal features of high-credibility neighbor vehicles. Quantitative and qualitative experimental results on two datasets, OPV2V and DAIRV2X, show that the proposed method is superior to eight benchmark methods in terms of multimodal fusion quality and multi-vehicle collaborative efficiency, indicating that it has wide applicability and scalability in complex environments. At the same time, ablation studies and visualization analysis reveal the strong robustness of the proposed method in real-world scenarios. Future work will continue to study the impact of adverse factors on multi-vehicle collaborative perception, such as transmission delay, limited computing power, etc. At the same time, improving the efficiency of model reasoning is an important issue worth exploring in actual deployment.

[0130] References

[0131] [1]Van Brummelen J,O'brien M,Gruyer D,et al.Autonomous VehiclePerception:The Technology of Today and Tomorrow[J].Transportation ResearchPart C:Emerging Technologies,2018,89:384-406.

[0132] [2]Dorri A,Kanhere SS,Jurdak R.Multi-agent Systems:A Survey[J].IEEEAccess,2018,6:28573-28593.

[0133] [3]Zhou Y, Xiao J, Zhou Y, et al. Multi-Robot Collaborative Perception with Graph Neural Networks[J]. IEEE Robotics and Automation Letters, 2022, 7(2): 2289-2296.

[0134] [4]Xu R,Tu Z,Xiang H,et al.CoBEVT:Cooperative Bird's Eye ViewSemantic Segmentation with Sparse Transformers[C].Conference on RobotLearning,2023:989-1000.

[0135] [5]Hu Y,Lu Y,Xu R,et al.Collaboration Helps Camera Overtake Lidar in3D Detection[C].Proceedings of The IEEE / CVF Conference on Computer Vision andPattern Recognition,2023:9243-9252.

[0136] [6]Lin C,Tian D,Duan X,et al.V2VFormer:Vehicle-to-Vehicle CooperativePerception with Spatial-Channel Transformer[J].IEEE Transactions onIntelligent Vehicles,2024.

[0137] [7]Wang T H,Manivasagam S,Liang M,et al.V2VNet:Vehicle-to-VehicleCommunication for Joint Perception and Prediction[C].European Conference onComputer Vision,2020:605-621.

[0138] [8]Xu R,Xiang H,Tu Z,et al.V2X-ViT:Vehicle-to-Everything CooperativePerception with Vision Transformer[C].European Conference on Computer Vision,2022:107-124.

[0139] [9]Zhang H,Luo G,Cao Y,et al.Multi-Modal Virtual-Real Fusion BasedTransformer for Collaborative Perception[C].IEEE International Symposium onParallel Architectures,Algorithms and Programming,2022:1-6.

[0140]

[10] Xiang H,Xu R,Ma J.HM-ViT:Hetero-Modal Vehicle-to-VehicleCooperative Perception with Vision Transformer[C].IEEE InternationalConference on Computer Vision,2023:284-295.

[0141]

[11] Yin H,Tian D,Lin C,et al.V2VFormer++:Multi-Modal Vehicle-to-Vehicle Cooperative Perception via Global-Local Transformer[J].IEEETransactions on Intelligent Transportation Systems,2023.

[0142]

[12] Liu Z,Tang H,Amini A,et al.BEVFusion:Multi-Task Multi-SensorFusion with Unified Bird's-Eye View Representation[C].IEEE InternationalConference on Robotics and Automation.IEEE,2023:2774-2781.

[0143]

[13] Vora S,Lang AH,Helou B,et al.PointPainting:Sequential Fusion for3D Object Detection[C].IEEE Conference on Computer Vision and PatternRecognition.2020:4604-4612.

[0144]

[14] Yin T,Zhou X, P.Multimodal Virtual Point 3D Detection[J].Advances in Neural Information Processing Systems,2021,34:16494-16507.

[0145]

[15] You Y,Wang Y,Chao W L,et al.Pseudo-LiDAR++:Accurate Depth for 3DObject Detection in Autonomous Driving[C].International Conference onLearning Representations(ICLR),2020.

[0146]

[16] Philion J,Fidler S.Lift,Splat,Shoot:Encoding Images fromArbitrary Camera Rigs by Implicitly Unprojecting to 3D[C].European Conferenceon Computer Vision,2020:194-210.

[0147]

[17] Wang C,Ma C,Zhu M,et al.PointAugmenting:Cross-Modal Augmentationfor 3D Object Detection[C].IEEE Conference on Computer Vision and PatternRecognition,2021:11794-11803.

[0148]

[18] Liang T,Xie H,Yu K,et al.BEVFusion:ASimple and Robust Lidar-Camera Fusion Framework[J].Advances in Neural Information Processing Systems,2022,35:10421-10434.

[0149]

[19] Bai X,Hu Z,Zhu X,et al.TransFusion:Robust Lidar-Camera Fusion for3D Object Detection with Transformers[C].IEEE Conference on Computer Visionand Pattern Recognition,2022:1090-1099.

[0150]

[20] Chen Q,Ma X,Tang S,et al.F-cooper:Feature based CooperativePerception for Autonomous Vehicle Edge Computing System Using 3D Point Clouds[C].IEEE Symposium on Edge Computing,2019:88-100.

[0151]

[21] Li Y,Ren S,Wu P,et al.Learning Distilled Collaboration Graph forMulti-Agent Perception[J].Advances in Neural Information Processing Systems,2021,34:29541-29552.

[0152]

[22] Xu R,Xiang H,Xia X,et al.OPV2V:An Open Benchmark Dataset andFusion Pipeline for Perception with Vehicle-to-Vehicle Communication[C].IEEEInternational Conference on Robotics and Automation,2022:2583-2589.

[0153]

[23] Cui J,Qiu H,Chen D,et al.Coopernaut:End-to-End Driving withCooperative Perception for Networked Vehicles[C].IEEE Conference on ComputerVision and Pattern Recognition,2022:17252-17262.

[0154]

[24] Liu Y C,Tian J,Ma C Y,et al.Who2com:Collaborative Perception viaLearnable Handshake Communication[C].IEEE International Conference onRobotics and Automation,2020:6876-6883.

[0155]

[25] Lang A H,Vora S,Caesar H,et al.PointPillars:Fast Encoders forObject Detection from Point Clouds[C].IEEE Conference on Computer Vision andPattern Recognition,2019:12697-12705.

[0156]

[26] Tan M,Le Q.Efficientnet:Rethinking Model Scaling forConvolutional Neural Networks[C].International Conference on MachineLearning,2019:6105-6114.

[0157]

[27] Zhu X,Su W,Lu L,et al.Deformable DETR:Deformable Transformers forEnd-to-End Object Detection[C].International Conference on LearningRepresentations,2021.

[0158]

[28] Liu W,Anguelov D,Erhan D,et al.SSD:Single Shot Multibox Detector[C].European Conference on Computer Vision,2016:21-37.

[0159]

[29] Ross T Y,Dollár G.Focal Loss for Dense Object Detection[C].IEEEConference on Computer Vision and Pattern Recognition,2017:2980-2988.

[0160]

[30] Ren S,He K,Girshick R,et al.Faster R-CNN:Towards Real-time ObjectDetection with Region Proposal Networks[J].IEEE Transactions on PatternAnalysis and Machine Intelligence,2016,39(6):1137-1149.

[0161]

[31] Yu H,Luo Y,Shu M,et al.DAIR-V2X:ALarge-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection[C].IEEE Conference on ComputerVision and Pattern Recognition,2022:21361-21370.

[0162]

[32] Liu Y C,Tian J,Glaser N,et al.When2com:Multi-Agent Perception viaCommunication Graph Grouping[C].IEEE Conference on Computer Vision andPattern Recognition,2020:4106-4115.

[0163]

[33] Hu Y,Fang S,Lei Z,et al.Where2comm:Communication-EfficientCollaborative Perception via Spatial Confidence Maps[J].Advances in NeuralInformation Processing Systems,2022,35:4874-4886。

Claims

1. A multi-vehicle collaborative perception method for modality loss in an autonomous driving scenario, which is applied to an autonomous driving scenario; in an autonomous driving scenario, vehicles continuously scan and monitor the surrounding environment through on-board laser radars and cameras to perceive and understand the environmental state; The steps of the multi-vehicle collaborative perception method include: (i) First, the BEV features obtained by the vehicle's laser radar and camera are fused to obtain the fused BEV feature B; (2) A multi-vehicle collaborative perception method based on a multi-head attention mechanism is used to select collaborative neighbor vehicles; (iii) Using a 3D object detection head to identify multiple objects around the vehicle; Its characteristic is that in step (ii), the correlation of multimodal features of adjacent vehicles is used to design a multi-vehicle collaborative perception strategy based on a multi-head attention mechanism on the constructed collaborative network, and the reputation value is evaluated using vehicle status and demand information to screen out collaborative neighbor vehicles with high reputation value; at the same time, multi-head attention is used to fuse the BEV features of collaborative vehicles; the collaborative fusion feature B of vehicle i i ' is used to input step (iii); Design a topology diagram first To build a multi-vehicle cooperative network, is the point set consisting of vehicle i, is the edge set connecting the vehicles, is the point set of neighboring vehicle j; The multi-vehicle collaborative perception strategy includes two steps: S1. Vehicle From its neighbor set The neighbor point set to be cooperated is selected according to the credibility value Its cardinality is J i ; S2, vehicle i and its selected neighbors perform multi-modal feature fusion based on multi-head attention; Vehicle i uses a set of convolutions to extract B i Compressed into query features with required information And broadcast to all neighboring vehicles; B i represents the multimodal fused BEV features extracted from vehicle i; Indicates the BEV feature B i Perform convolution operation; Neighbor j generates B using a set of convolutions i,j Key information carried Combined with the query feature ε i Calculating the reputation of dot product attention in The parameter matrix representing the dot product attention, B i,j represents the multimodal fused BEV features of neighbor j that cooperates with vehicle i; represents the BEV feature B of the jth neighbor of vehicle i i,j Perform convolution operation; When all neighbors send back reputation values, vehicle i normalizes this set of reputation values ​​using the Softmax function, i.e. And select the probability value greater than the preset threshold neighbors as selected cooperative vehicles, Vehicle i sends a collaboration request to the selected collaborative vehicles, which immediately respond and send B after receiving the request information. i,j Give vehicle i; Vehicle i receives the collaborative feature set After that, vehicle i and its neighbor j perform feature map fusion based on multi-head attention, by setting the expression of an attention head h as in and represents the parameter matrix of the linear layer, represents the attention weight matrix for calculating the dot product of the query vector and the key vector; represents the parameters of the attention head h corresponding to the jth neighbor of vehicle i, represents the parameters of the attention head h corresponding to vehicle i, and C represents the number of channels; The multi-head attention corresponding to the cooperative vehicle is MultiHeadATT i,j =[head i,j,1 ,head i,j,2 …,head i.j,h ] The multi-head attention weight based on the normalized reputation value IN i,j =MultiHeadATT i,j ·γ i,j Based on this, the collaborative fusion features of vehicle i are generated where FFN(·) is a feed-forward neural network consisting of a set of linear layers and activation functions.

2. The multi-vehicle collaborative perception method for modality loss in an autonomous driving scenario according to claim 1, characterized in that Step (a) includes: step 1) BEV feature extraction; step 2) multimodal feature fusion; In step 1), extract the BEV feature B of the laser radar (L) And the camera's BEV feature B (C) ; Among them, the BEV feature of the camera is (C) The depth information provided by the LiDAR is projected onto the RGB image acquired by the camera, and the extracted RGB image depth information is converted into the BEV space; if the image pixel acquired by the camera does not have the corresponding LiDAR projected depth, the RGB image depth is estimated; In step 2), the BEV feature B of the lidar (L) and the camera's BEV feature B (C) B is obtained based on the fusion of cross-modal deformable attention layer and self-attention layer.

3. The multi-vehicle collaborative perception method for modality loss in an autonomous driving scenario according to claim 2, The method is characterized in that in step 1), for each vehicle, the BEV features of the two modalities of laser radar and camera are extracted respectively, specifically: a dual-branch network of laser radar and camera is used to extract the BEV features of the two modalities; Extracting BEV features of LiDAR (L) The method is that the vehicle uses PointPillars to extract point cloud features from the data collected by the lidar. The steps include: First, the point cloud data is divided into H×W grids according to the two-dimensional plane; the point cloud data falling into the same grid is regarded as in a Pillar, and a point in the pillar is represented by a vector l1=(x, y, z, r), where (x, y, z) represents the position information and r represents the reflection intensity; Consider the geometric center of Pillar l2=(x c ,y c ,z c ) and the center offset l3=(x p ,y p )=(xx c ,yy c ), the point cloud is encoded into a 9-dimensional vector When the vehicle regularizes the number of points in Pillar to T through random sampling and zero padding, the dimension of the point cloud becomes H×W×T×9; H and W represent the number of grid rows and columns respectively. Next, the vehicle uses the PointNet network to extract the intermediate features of the point cloud Then the BEV features of the lidar are output through maximum pooling Where N is the number of feature channels; Extract the BEV feature of the laser camera (C) The method has two data-aware cases, namely Case 1 and Case 2, where: Case 1: When the LiDAR and camera simultaneously obtain information about each point in the physical space, the vehicle directly maps the LiDAR coordinates to the image pixel coordinates, thereby obtaining the image depth information. The mapping process undergoes multiple coordinate transformations, including: transformation from the lidar coordinate system to the camera coordinate system, transformation from the camera coordinate system to the image coordinate system, and transformation from the image coordinate system to the pixel coordinate system; The LiDAR coordinate system is regarded as the world coordinate system, where the world coordinate of any point is (X W ,Y W ,Z W ) T ; Let the transformation matrix E from the laser radar coordinate system to the camera coordinate system be the camera's external parameter matrix. The point (X W ,Y W ,Z W ) T In the camera coordinate system, it is expressed as Where (X W ,Y W ,Z W ,1) T and (X C ,Y C ,Z C ,1) T All represent homogeneous coordinates; Assume the focal length of the camera is f, point (X C ,Y C ,Z C ) T is transformed into the image coordinate system, i.e. Where (x,y,1) T represents homogeneous coordinates, K represents the intrinsic parameter matrix of the camera; Each pixel of the camera corresponds to a photosensor. The physical dimensions of the matrix centered on this photosensor are dx and dy respectively. The discrete image coordinate system is used to establish a pixel coordinate system with the upper left corner as the origin. The point (x, y) T is converted to the pixel coordinate system, that is, Formula (3) gives the pixel coordinate system representation of the camera image, denoted as In summary, the distance Z from the pixel point (u, v) to the camera plane is C That is the image depth information d; Case 2: When the image pixels acquired by the camera do not have the corresponding LiDAR projection depth, the vehicle still needs to obtain reliable depth information to extract BEV features. The LSS algorithm is used to implicitly estimate the image depth: The LSS algorithm generates a discrete point set of D depths for each pixel (u, v) Corresponding to D 3D points The 3D point set generates D planes from the camera plane, so the image is represented as a frustum point cloud Next, EfficientNet is used to extract the D-dimensional depth features and N-dimensional semantic features of each pixel in the image; The projected depth d calculated in Case 1 is mapped to a discrete depth value and combined with the depth feature predicted by the Softmax function. The outer product operation is then performed with the semantic feature to obtain the probability distribution of the point (u, v, d) and the image feature point cloud. Finally, calculate the 3D coordinates of the pixel (u, v) in the vehicle coordinate system (x,y,z) T =d·(K·E) -1 ·(u,v,1) T Refer to the PointPillar method, image feature point cloud F (C) According to formula (4), they are assigned to different Pillars, and the image feature point clouds belonging to the same Pillar are summed and pooled to output the BEV feature map This feature map represents the image features around the central vehicle from a bird's eye view.

4. The multi-vehicle collaborative perception method for modality loss in an autonomous driving scenario according to claim 2, characterized in that In step 2), sine and cosine functions are used as query features Generate positional encoding And P e Add Q element by element to get a query feature containing spatial location information At the same time, the spliced ​​lidar and camera BEV features V = (B (C) ,B (L) ) as the value feature, where B (C) or B (L) Allow missing; Set a scaling factor r to generate H on the value feature V. p ×W p A grid of reference points and the value of any reference point p is a linearly spaced two-dimensional coordinate {(0,0),…,(H p -1,W p -1)}, where H p =H / r,W p =W / r; To describe relative positions, these reference points are normalized to [-1, +1], where (-1, -1) represents the upper left corner and (+1, +1) represents the lower right corner; The deformable attention mechanism divides the feature map into M groups in the channel dimension to form M attention heads; each attention head only focuses on the S key sampling points around the reference point p and calculates their value features V; In order to determine the locations of these key sampling points, the query vector Q′ is input into two linear layer modules in parallel; Q′ passes through the lidar linear layer (Linear-L) and the camera linear layer (Linear-C) Generates a position offset relative to the reference point p Where Concat(·) represents the channel concatenation operation; When B (C) or B (L) When missing, Q′ only passes through the linear layer of the non-missing mode and no concatenation operation is performed; In the sampling offset module, the reference point p is added to Δp to obtain the sampling position of the value feature V; the coordinate value of p+Δp may be non-integer, so the following formula selects the four points closest to p on V (a x,l ,a y,l ), l=1,…,4 perform bilinear interpolation to calculate the eigenvalues ​​of S key sampling points And get the sampled value features Where g(a,b)=max(0,1-|ab|), a and b represent two independent variables, and the subscripts x and y represent the x and y axes of the plane coordinates, respectively; Multi-head attention is calculated using query, key, value, and position offset, where the output of the mth attention head is At the same time, the query vector Q′ passes through the camera linear layer and the lidar linear layer Get splicing features When B (C) or B (L) When missing, Q′ only passes through the linear layer of the non-missing mode and no concatenation operation is performed; Next, stitching features Input into cross-modal deformable attention through Softmax function; The multi-head attention of the sampling bias module also designs M attention heads and generates cross-modal deformable attention weights A, which are used to scale features. To obtain cross-modal query features in and represents a learnable projection matrix; Multiple self-attention layers are added after multiple cross-modal deformable attention layers; the query feature and value feature of the self-attention layer are both Q″, so only a single-modal linear layer is used to generate Δp and A when constructing the self-attention layer. Therefore, the value feature V in the above formula is replaced by Q″ to calculate the self-attention Thus, the vehicle uses the cross-modal deformable attention layer and the self-attention layer to establish the relationship between multimodal features and obtain the final 5. The multi-vehicle collaborative perception method for modality loss in an autonomous driving scenario according to claim 1, characterized in that In step (iii), in the 3D target detection head, a 3D target detection model is used to identify multiple targets around the vehicle.

6. The multi-vehicle collaborative perception method for modality loss in an autonomous driving scenario according to claim 5, characterized in that In the 3D detection head, vehicle i will collaborate to fuse feature B i ' Input the 3D detection head to observe the objects around the vehicle body; the 3D detection head uses the SSD model to predict the 3D bounding box and the confidence of the target category; SSD includes a classification head and a regression head, where the classification head predicts the confidence of the object category in each predicted box, and the regression head predicts the offset of each predicted box relative to the real box.

7. The multi-vehicle collaborative perception method for modality loss in an autonomous driving scenario according to claim 6 is characterized in that when the SSD model is trained, the target classification loss is used. and regression loss To optimize its own parameters, the steps include: First, given a real box vector (x, y, z, ξ,τ,θ), where (x,y,z) represents the target center, represents the size of the 3D box, and θ represents the rotation angle of the target; Then use the following Focal los as the target classification loss to balance the foreground target and background area Where μ represents the class probability, α1 and α2 are hyperparameters; Define the regression loss between the predicted value and the true value Where β is an adjustable parameter; Smooth L1 The loss is used to supervise the size of the 3D target box Where θ′ represents the rotation angle of the prediction box; The total loss of model training is Where λ1 and λ2 represent the classification loss coefficient and regression loss coefficient respectively.

Images