Automatic driving multi-sensing data self-supervision pre-training and fusion sensing method

Abstract

The invention discloses an automatic driving multi-sensing data self-supervision pre-training and fusion sensing method, and relates to the field of image-point cloud sensing data fusion, multi-modal feature self-supervision pre-training and representation reconstruction and multi-source information fusion. Forming a three-dimensional point cloud voxel representation and image block, and generating a feature embedding vector; a feature mask set is obtained through random mask operation, and the robustness of feature learning is improved; through a multi-sensing data interaction module and space projection transformation, scene reconstruction is carried out by using reconstruction branches, original features are recovered, and optimal pixel-point cloud fusion features are learned in a self-supervision mode; and finally, designing an image-point cloud fusion perception algorithm based on a proxy attention mechanism, bridging global context information through proxy marks, fully mining multi-sensor data complementation characteristics, and improving heterogeneous information fusion effects and environmental perception performance.

Description

Technical Field

[0001] This application relates to the technical fields of image-point cloud sensing data fusion, multi-modal feature self-supervised pre-training and representation reconstruction, and multi-source information fusion, and particularly relates to a self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving. Background Art

[0002] Autonomous driving marks a transformative leap in the transportation field, bringing potential significant improvements in aspects such as safety, efficiency, and accessibility. The key to this progress lies in the vehicle's ability to interpret its surrounding environment. To cope with the complexity of real-world scenarios, integrating multiple sensors is crucial: cameras provide detailed visual information, while lidar provides precise geometric data. Through this multi-sensor fusion, a comprehensive understanding of the environment can be achieved. How to efficiently learn the representation information of different sensing data and perform accurate point-to-point fusion remains an important challenge in the current field.

[0003] On the one hand, masked autoencoders have shown good performance in learning general data representations and reducing the dependence on labeled data. However, this method faces serious geometric distortion problems when projecting lidar data onto the camera plane. This distortion stems from the inconsistent spatial distribution of the point cloud; in pixel coordinates, points in the point cloud that are physically far apart or close may be adjacent to each other, making it difficult to partition and mask these merged image and point cloud blocks. Due to different sensor settings, lidar can capture point clouds outside the camera's field of view. For example, the Waymo Open Dataset provides a 360° lidar view, which results in a large amount of lidar point clouds that cannot be projected. At the same time, considering the sparsity of lidar data itself, a large amount of information is lost during registration alignment, severely restricting the effectiveness and accuracy of feature fusion.

[0004] The bird's-eye view, as a commonly used representation method, can align point cloud data with visual features through depth projection and transform it into a unified bird's-eye view space to preserve geometric structure and semantic density. However, the bird's-eye view space lacks height information, which can lead to inaccurate descriptions of object shapes and positions and is difficult to provide sufficient three-dimensional scene information for autonomous driving perception.

[0005] On the other hand, the multi-source feature fusion method has been a hot topic in recent years. The proposal-level fusion method generates proposals in three-dimensional space, projects them onto images to extract Region of Interest (RoI) features and then fuses them; the point-level fusion method usually maps image semantic features to foreground LiDAR point clouds for data or feature-level fusion. However, due to the different densities of the two modalities, there is information loss in the projection from the camera to LiDAR semantically. The fusion method based on the Transformer network mainly aggregates features by calculating the similarity between each pair of query-key through Softmax attention. However, this attention mechanism leads to high computational costs and is difficult to be applied on a large scale in vision tasks. Existing work attempts to reduce computational complexity by designing efficient attention patterns: Swin Transformer reduces computational complexity by reducing the receptive window size; PVT confines self-attention calculation to local windows and adopts a sparse attention pattern to relieve the computational burden. Although these methods effectively reduce the computational pressure by reducing the number of keys and values, Swin Transformer limits the range of information that the model can directly capture, making it difficult for the model to capture long-range dependencies beyond the window size; although the sparsity of PVT reduces the computational amount, it also reduces the amount of information that the model can utilize, thus affecting the modeling of long-range relationships. Inevitably, they damage the model's ability to model long-range relationships, and the damage to the modeling ability may directly affect the performance of the model; the localized attention mechanism limits the generalization ability of the model to specific datasets or tasks. When facing new data with different feature distributions or more complex scenarios, the model may be difficult to adapt and exhibit good performance.

[0006] Generally speaking, the fusion of multi-source data (i.e., multi-sensor data) in the field of autonomous driving still faces the following key challenges: how to construct a unified self-supervised learning framework to fully understand and mine the characteristics of multi-source heterogeneous data and form effective multi-modal representations, and how to design accurate and efficient multi-modal feature fusion methods to promote robust environmental perception capabilities. These problems limit the performance of existing methods in complex scenarios. Therefore, there is an urgent need for an innovative solution that can achieve joint feature representation, accurate information alignment, and robust perception to promote the further development of autonomous driving perception technology. Summary of the Invention

[0007] The purpose of this application is to provide a self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving to solve the problems of insufficient understanding of mining the characteristics of multi-sensor data and poor environmental perception ability.

[0008] To achieve the above purpose, this application provides the following solutions: The present application provides a self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving, including: Process the multi-sensor data to obtain voxelized point cloud data and image patches, and embed the voxelized point cloud data and the image patches into tokens to obtain voxelized point cloud data tokens and image data tokens; the multi-sensor data includes lidar point cloud data and image data.

[0009] Mask the voxelized point cloud data tokens and the image data tokens to obtain image features and point cloud features, and construct a masked token set.

[0010] Project the image features and point cloud features in the token set to obtain the positions of the image features and point cloud features on the camera view image.

[0011] Based on the multi-sensor data interaction module, fuse the image features and point cloud features according to the positions of the image features and point cloud features on the view image to obtain fused features.

[0012] Back-project the fused features, and perform scene reconstruction based on the image feature branch and the point cloud feature branch to obtain a new scene.

[0013] Based on the new scene, bridge the global information of the new scene through proxy tokens to achieve the fusion of multi-sensor data.

[0014] According to the specific embodiments provided by the present application, the present application has the following technical effects: The present application provides a self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving. By processing image-lidar multi-sensor data, three-dimensional point cloud voxel representations and image patches are formed, and feature embedding vectors are generated; random masking is performed on the point cloud voxels and image features to simulate situations such as sensor failures, external noises, and signal losses in the real world, enhancing the robustness of feature learning; based on the multi-sensor data interaction module (Multi-sensor Interaction Module, MIM), according to the relative position relationship in the image-point cloud feature space, a joint representation with rich semantic information is formed through projection transformation, and scene reconstruction is performed separately through the image and point cloud branches to learn the optimal image-point cloud fusion features in a self-supervised manner. Finally, an image-point cloud feature fusion perception algorithm based on the proxy attention mechanism is designed to fuse the global context scene information with the proxy tokens as the bridge, fully exploiting the complementary characteristics of multi-sensor data, and improving the heterogeneous information fusion effect and environmental perception performance. Description of the Drawings

[0015] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0016] Figure 1 It is a schematic flowchart of the self-supervised pre-training and fusion perception method for multi-sensor data in the embodiments of the present application; Figure 2 It is a schematic working diagram of the self-supervised pre-training and fusion perception method for multi-sensor data provided in the embodiments of the present application; Figure 3 It is a schematic diagram of three attention mechanisms provided in the embodiments of the present application. Detailed implementation manners

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0018] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will further describe the present application in detail with reference to the drawings and specific implementation manners.

[0019] As Figure 2 shown, the present application discloses a self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving, aiming to promote the interactive sharing of different sensing features and achieve the efficient fusion of image semantic features and point cloud geometric information.

[0020] On the one hand, a self-supervised pre-training framework for multi-sensor data is constructed to realize the extraction and learning of general representations of multi-source data: First, camera and radar data features are respectively extracted and projected into an enhanced three-dimensional volume space (Volume) to make up for the lack of height information in the bird's-eye view (BEV) perspective, and a self-supervised pre-training model for camera-radar features based on the mask mechanism is proposed; Subsequently, a multi-sensor data interaction module (Multi-sensor InteractionModule, MIM) is designed to promote the efficient communication and full fusion of different sensing data information.

[0021] On the other hand, the multi-sensor feature fusion perception method based on the proxy attention mechanism in this application uses proxy agents as an intermediate medium to enhance the correlation between queries and key-value features, alleviate the interference of irrelevant noise on the fusion process and computational latency, and efficiently improve the accuracy of multi-source information fusion and the robustness of perception.

[0022] This application gives full play to the potential of masked self-supervised learning, significantly reducing the dependence on a large amount of labeled data; efficiently representing image and point cloud data through a unified three-dimensional voxel space (Volume) representation, using a multi-sensor interaction module (Multi-sensor Interaction Module, MIM) to achieve deep interaction and information enhancement of image-point cloud features, and combining the proxy attention fusion perception method to effectively solve the problems existing in traditional methods such as difficulty in aligning multi-sensor data, loss of modal information, and poor fusion effect, while greatly improving the accuracy and robustness of three-dimensional space target perception in the autonomous driving scenario.

[0023] As Figure 1 shown, the embodiment of this application provides a self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving, which specifically includes: S1: Process the multi-sensor data to obtain voxelized point cloud data and image patches, and embed the voxelized point cloud data and the image patches as tokens to obtain voxelized point cloud data tokens and image data tokens; the multi-sensor data includes lidar point cloud data and image data.

[0024] S2: Mask the voxelized point cloud data tokens and the image data tokens to obtain image features and point cloud features, and construct a masked token set.

[0025] S3: Project the image features and point cloud features in the token set to obtain the positions of the image features and point cloud features on the camera view image.

[0026] S4: Based on the multi-sensor interaction module, fuse the image features and point cloud features according to the positions of the image features and point cloud features on the view image to obtain fused features.

[0027] S5: Back-project the fused features and perform scene reconstruction based on the image feature branch and the point cloud feature branch to obtain a new scene.

[0028] S6: Based on the new scene, bridge the global information of the new scene through proxy tokens to achieve the fusion of multi-sensor data.

[0029] Further, in an exemplary embodiment, step S1 can be replaced by the following steps.

[0030] S101: Map the three-dimensional spatial point positions in the lidar point cloud data to a voxel grid to obtain voxelized point cloud data.

[0031] Perform voxelization processing on the input lidar point cloud data. Each three-dimensional spatial point in the point cloud is mapped to a discrete voxel grid according to its position, thereby generating a voxelized point cloud feature representation.

[0032] S102: Embed the voxelized point cloud data into tokens to obtain the voxelized point cloud data tokens.

[0033] The voxelized point cloud data is then embedded into tokens for subsequent feature extraction. At the same time, the input image data is segmented. The image data is divided into non-overlapping image patches of a fixed size, and position encoding is added to each image patch to be embedded into tokens.

[0034] S103: Segment the image data to obtain non-overlapping image patches of a preset size.

[0035] S104: Add position encoding to the non-overlapping image patches to obtain image data tokens.

[0036] This position encoding retains the relative position information of the image patches in the two-dimensional space, enabling the subsequent model to accurately capture the spatial relationships of the image patches.

[0037] Further, in an exemplary embodiment, step S2 can be replaced by the following steps.

[0038] S201: Use a random masking strategy to mask the voxelized point cloud data tokens and the image data tokens to obtain image features and point cloud features, and construct a masked token set.

[0039] After generating the tokens of the image and the point cloud, a random masking strategy is adopted to partially mask the tokens of the two modalities. This masking mechanism constructs a masked token set by randomly selecting a part of the tokens for hiding ( , ), where ( , ) is the set of features of the masked image I and the point cloud voxel V. The masking operation can not only simulate data missing in the real environment but also force the model to learn to infer the information of the hidden part from the unmasked tokens, thereby improving the self-supervised learning ability of the model.

[0040] Further, in an exemplary embodiment, step S3 can be replaced by the following steps.

[0041] S301: Based on the ego-vehicle coordinate system, map each voxel of the point cloud feature to the corresponding position in the 3D volume space to obtain the projection of the point cloud feature.

[0042] Project the image and the point cloud feature into a unified 3D volume space to achieve geometric and semantic fusion and alignment. The projection of the point cloud feature is based on the ego-vehicle coordinate system and is achieved by mapping each voxel to the corresponding position in the 3D volume space. The spatial information of the voxelized point cloud is referenced to the vehicle, and the three-dimensional geometric attributes of the point cloud data are used to accurately embed it into the 3D volume space, thereby constructing a spatially consistent representation of the point cloud feature. This projection method ensures that the geometric accuracy of the point cloud data is not affected by any distortion.

[0043] S302: Use the formula to obtain the projection of the image feature; the positions of the image feature and the point cloud feature on the camera view image include the projection of the point cloud feature and the projection of the image feature.

[0044] where is the image feature; i represents the camera view image; j represents the 3D reference point; is the total number of points queried for each 3D voxel; is the feature value of the j-th point in the i-th camera view; is the projection function for obtaining the position of the j reference point on the i-th view image, p is the projection of the 3D point p onto the image plane of the i-th perspective to obtain the position of the point on the image; means that the 3D point is only associated with a specific view during the projection process; is the set of query points for each 3D voxel.

[0045] For the image feature , a 2D-3D spatial cross-attention mechanism is used to complete the projection. Each image token queries its corresponding position in the 3D volume space, and the corresponding 3D points are projected into the 2D view using the camera's intrinsic and extrinsic parameters. The 3D points are only associated with a specific view during the projection process, called view . The 2D features of the 2D view are sampled at the positions of these projected 3D reference points from view .

[0046] where is the set of query points for the 3D voxel (volume), ∈ is the number of channels; H is the height of the voxel in three-dimensional space; W is the width of the voxel in three-dimensional space; Z is the depth of the voxel in three-dimensional space.

[0047] Furthermore, in an exemplary embodiment, step S4 can be replaced by the following steps.

[0048] S401: Concatenate the image features and the point cloud features to form a query vector.

[0049] Furthermore, a multi-sensor data interaction module is proposed to facilitate the efficient fusion of two types of 3D voxel features and . Considering the high-resolution and high-performance requirements of downstream tasks and the limitation of the token sequence scale in the standard self-attention mechanism, deformable self-attention is adopted here for fusion calculation to reduce the computational requirements. Each module consists of deformable self-attention, a feed-forward network, and normalization: Concatenate and along the channel dimension to form a query vector .

[0050] S402: Input the query vector into the multi-sensor data interaction module for feature interaction to obtain multi-modal interaction features; the multi-modal interaction features are fusion features.

[0051] Input into the MMIM for feature interaction to form multi-modal interaction features , and the specific process is as follows: .

[0052] Among them, quantity, is the total number of sampled keys. and respectively represent the offset and attention weight of the -th sampling point in the -th attention head. The value range of is [0, 1], and it is normalized by . Finally, is split along the channel dimension to obtain 3D voxel features of a specific modality (

[0053] Furthermore, in an exemplary embodiment, step S5 can be replaced by the following steps.

[0054] S501: For the lidar branch, project back to the 3D volume space according to the sampling positions of each voxel of the mask at the position of the point cloud features on the view image in the ego-vehicle coordinate system to obtain the corresponding 3D features; the lidar branch is the point cloud feature branch.

[0055] Based on consistent 3D voxel features, the fused features can be efficiently and concisely projected back to the 2D view and 3D volume space respectively. For the LiDAR branch, according to the sampling positions of the masked voxels in the ego-vehicle coordinate system, the corresponding 3D features can be projected back to the original space, and these features will contain rich image semantic information after fusion.

[0056] S502: Based on the LiDAR branch, use to reconstruct the point cloud pixels; where is the chamfer loss function; is the Chamfer distance function; is the voxel decoder; is the projected voxel feature; is the masked LiDAR point cloud divided into voxels.

[0057] S503: For the camera branch, use the projection function to project the 3D voxel features at the positions of the image features on the view image to the 2D plane features; the camera branch is the image feature branch.

[0058] Use the projection function to project the 3D voxel features to the 2D view features at the corresponding coordinate positions . The projection function is defined as follows: .

[0059] Where is the position in the 3D voxel, and are the intrinsic matrix and extrinsic matrix of the camera respectively, where is the 4×4 matrix space.

[0060] S504: Based on the image feature branch, use the formula to reconstruct the image pixels; where is the loss of image reconstruction, is the mean squared error loss MSE, is the original image in the pixel space; is the image decoder; [[ID=5�]] is the projected image feature.

[0061] For the scene reconstruction task, the image and point cloud branches need to be considered separately. On the one hand, the mean squared error (MSE) loss is used as the supervision signal for image pixel reconstruction. On the other hand, the Chamfer distance is used to measure the difference between two point sets at different scales for point cloud voxel reconstruction, and the Chamfer loss Expressed as: .

[0062] Wherein, is the Chamfer distance function; is the voxel decoder; is the predicted result of the scene voxel; the cumulative voxel reconstruction loss is defined as: .

[0063] S505: Obtain a new scene according to the point cloud pixels and the image pixels.

[0064] Furthermore, in an exemplary embodiment, as Figure 3 shown, step S6 can be replaced by the following steps.

[0065] S601: Use a formula to bridge the global information of the query vector and the key-value vector to achieve the fusion of multi-sensor data; wherein, is the Agent attention mechanism; A is the agent , is dimensional matrix space; is the Softmax attention mechanism; Q is the query value of the query vector; K is the key value of the key-value vector; V is the value matrix.

[0066] Considering the heterogeneous characteristics of image and point cloud features, it is difficult for current mainstream attention methods to achieve accurate and efficient fusion effects: Softmax attention needs to calculate the similarity between all queries (Q) and key-value pairs (K / V), and its computational cost is very high; while linear attention uses a mapping function to adjust the order of QKV vectors to reduce the computational complexity, but this method restricts the feature expression ability. The expressions of the two attention mechanisms are as follows: .

[0067] .

[0068] Wherein, is Softmax; is the Linear attention mechanism; Q is the query value of the query vector; K is the key value of the key-value vector; V is the value matrix; represents the Softmax function.

[0069] Therefore, a multi-source data fusion perception method based on the agent attention mechanism is proposed here, which elegantly bridges the global information of Q and K / V vectors through a set of agent tokens (Agent Token).

[0070] The calculation process of proxy attention includes two parts: Agent Aggregation and Agent Broadcast. Among them, is Agent Aggregation; is Agent Broadcast. Specifically, the proxy tokens are initially regarded as query tokens, and and are used to aggregate global features ; Subsequently, taking as keys and values, and calculating attention again with the query matrix , the proxy features obtained from the global are broadcast back to each query token to form the final output . This method avoids directly calculating the pairwise similarity between and , and at the same time realizes the efficient exchange of each pair of feature information. In the process of image-point cloud feature fusion, the point cloud features are used as queries (Q), the image features are used as key-value vectors (K / V), and the fusion result of the previous frame is used as the proxy token. Different modality-specific information and scene context relationships are iteratively fused from the temporal and spatial dimensions to form comprehensive and robust multi-sensor fusion features for downstream perception tasks.

[0071] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0072] In this article, specific examples are used to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. An autonomous driving multi-sensor data self-supervised pre-training and fusion perception method, characterized in that, The self-supervised pre-training and fusion perception method for multi-sensor data of autonomous driving includes: Processing multi-sensor data to obtain voxelized point cloud data and image patches, and embedding the voxelized point cloud data and the image patches into tokens to obtain voxelized point cloud data tokens and image data tokens; the multi-sensor data includes lidar point cloud data and image data; Masking the voxelized point cloud data tokens and the image data tokens to obtain image features and point cloud features, and constructing a masked token set; Projecting the image features and point cloud features in the token set to obtain the positions of the image features and point cloud features on the camera view image; Based on the multi-sensor data interaction module, fusing the image features and point cloud features according to the positions of the image features and point cloud features on the view image to obtain fused features; Back-projecting the fused features and performing scene reconstruction based on the image feature branch and the point cloud feature branch to obtain a new scene; Based on the new scene, bridging the global information of the new scene through proxy tokens to achieve the fusion of multi-sensor data.

2. The self-supervised pre-training and fusion perception method for multi-sensor data of an autonomous vehicle according to claim 1, wherein, Processing multi-sensor data to obtain voxelized point cloud data and image patches, and embedding the voxelized point cloud data and the image patches into tokens to obtain voxelized point cloud data tokens and image data tokens, specifically including: Mapping the three-dimensional spatial point positions in the lidar point cloud data into a voxel grid to obtain voxelized point cloud data; Embedding the voxelized point cloud data into tokens to obtain the voxelized point cloud data tokens; Performing block processing on the image data to obtain non-overlapping image patches of a preset size; Adding position encoding to the non-overlapping image patches to obtain image data tokens.

3. A self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving according to claim 1, characterized in that Masking the voxelized point cloud data tokens and the image data tokens to obtain image features and point cloud features, and constructing a masked token set, specifically including: Using a random masking strategy to randomly mask the voxelized point cloud data tokens and the image data tokens respectively to obtain image features and point cloud features, and constructing a masked token set.

4. A self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving according to claim 1, characterized in that, Projecting the image features and point cloud features in the token set to obtain the positions of the image features and point cloud features on the camera view image, specifically including: Based on the vehicle coordinate system, each voxel of the point cloud feature is mapped to the corresponding position in the 3D volume space to obtain the projection of the point cloud feature; using the formula , the projection of the image feature is obtained; Among them, is an image feature; i represents the camera view image; j represents the 3D reference point; is the total number of points queried for each 3D voxel; is the feature value of the j-th point in the i-th camera view; is a projection function used to obtain the position of the j reference point on the i-th view image. p projects the 3D point p onto the image plane of the i-th perspective to obtain the position of the point on the image; means that in the projection process, the 3D point is only associated with a specific view is the set of query points for each 3D voxel.

5. A self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving according to claim 1, characterized in that, Based on the multi-sensor data interaction module, fusing the image features and point cloud features according to the positions of the image features and point cloud features on the view image to obtain fused features, specifically including: Concatenating the image features and point cloud features to form a query vector; Inputting the query vector into the multi-sensor data interaction module for feature interaction to obtain multi-modal interaction features; the multi-modal interaction features are fused features.

6. The self-supervised pre-training and fusion perception method for multi-sensor data of an autonomous driving vehicle according to claim 1, characterized in that Back-projecting the fused features and performing scene reconstruction based on the image feature branch and the point cloud feature branch to obtain a new scene, specifically including: For the lidar branch, according to the sampling positions of each voxel of the mask of the position of the point cloud feature on the view image in the ego-vehicle coordinate system, projecting back to the 3D volume space to obtain corresponding 3D features; the lidar branch is the point cloud feature branch; Based on the lidar branch, using to reconstruct point cloud pixels; where is the chamfer loss function; is the Chamfer distance function; is the voxel decoder; is the projected voxel feature; is the masked LiDAR point cloud divided into voxels; For the camera branch, the projection function is used to project the 3D voxel features of the position of the image features on the view image into 2D plane features; the camera branch is an image feature branch; the 3D voxel features are 3D features; Based on the image feature branch, using the formula to reconstruct the image pixels; where is the loss of image reconstruction, is the mean squared error loss MSE, is the original image in the pixel space; is the image decoder; is the projected image feature; Based on the point cloud pixels and the image pixels, a new scene is obtained.

7. A self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving according to claim 1, characterized in that Based on the new scene, the global information is bridged through proxy tags to achieve the fusion of multi-sensor data, specifically including: Use the formula to bridge the global information of the query vector and the key-value vector to achieve the fusion of multi-sensor data; among them, is the Agent attention mechanism; A is the agent , is the dimensional matrix space; is the Softmax attention mechanism; Q is the query value of the query vector; K is the key value of the key-value vector; V is the value matrix.

Images