Self-supervised pre-training and fusion perception method for automatic driving multi-sensor data

Abstract

The application discloses an automatic driving multi-sensing data self-supervision pre-training and fusion perception method, relates to the fields of image-point cloud sensing data fusion, multi-modal feature self-supervision pre-training and representation reconstruction and multi-source information fusion, and the method processes image-radar multi-sensing data, forms three-dimensional point cloud voxel representation and an image block, and generates a feature embedding vector; a feature mask set is obtained through a random mask operation, and the robustness of feature learning is improved; through a multi-sensing data interaction module and spatial projection transformation, a scene is reconstructed by using a reconstruction branch, original features are recovered, and optimal pixel-point cloud fusion features are learned in a self-supervision mode; finally, an image-point cloud fusion perception algorithm based on a proxy attention mechanism is designed, global context information is bridged through a proxy label, complementary characteristics of multi-sensor data are fully tapped, and the effect of heterogeneous information fusion and the environmental perception performance are improved.

Description

Technical Field

[0001] The present application relates to the technical fields of image-point cloud sensor data fusion, multimodal feature self-supervised pre-training and representation reconstruction, and multi-source information fusion, and in particular to a method for self-supervised pre-training and fusion perception of multi-sensor data for autonomous driving. Background Art

[0002] Autonomous driving marks a transformative leap forward in transportation, offering potentially significant improvements in safety, efficiency, and accessibility. Key to this progress lies in the vehicle's ability to interpret its surroundings. To cope with the complexity of real-world scenarios, integrating multiple sensors is crucial: cameras provide detailed visual information, while lidar provides precise geometric data. This multi-sensor fusion enables a comprehensive understanding of the environment. However, efficiently learning representations of disparate sensor data and accurately fusing them in a point-to-point manner remains a key challenge in the field.

[0003] On the one hand, masked autoencoders have shown good performance in learning universal representations of data and reducing dependence on labeled data. However, this method faces serious geometric distortion problems when projecting lidar data onto the camera plane. This distortion stems from the inconsistency of the intrinsic spatial distribution of the point cloud; in pixel coordinates, point clouds that are physically far apart or close together may be adjacent to each other, making it difficult to divide and mask these merged images and point cloud blocks. Due to different sensor settings, lidar can capture point clouds outside the camera's field of view. For example, the Waymo Open Dataset provides a 360° lidar view, which results in a large number of lidar point clouds being unable to be projected. At the same time, considering the sparsity of the lidar data itself, a large amount of information is lost during registration and alignment, which seriously restricts the effectiveness and accuracy of feature fusion.

[0004] Bird's-eye view, a commonly used representation method, aligns point cloud data with visual features through depth projection and transforms it into a unified bird's-eye view space to preserve geometric structure and semantic density. However, the lack of height information in the bird's-eye view space leads to inaccurate descriptions of object shape and position, making it difficult to provide sufficient 3D scene information for autonomous driving perception.

[0005] On the other hand, multi-source feature fusion methods have been a hot topic in recent years. Proposal-level fusion methods generate proposals in three-dimensional space, project them onto the image, extract region-of-interest (RoI) features, and fuse them. Point-level fusion methods typically map image semantic features to foreground LiDAR point clouds for data or feature-level fusion. However, due to the difference in density between the two modalities, the camera-to-LiDAR projection suffers from semantic information loss. Transformer network-based fusion methods primarily achieve feature aggregation by calculating the similarity between each query-key pair using Softmax attention. However, this attention mechanism incurs high computational costs, making it difficult to apply on a large scale in vision tasks. Existing work attempts to reduce computational complexity by designing efficient attention models: the Swin Transformer reduces computational complexity by reducing the size of the receiving window; PVT alleviates the computational burden by limiting self-attention calculations to a local window and adopting a sparse attention model. Although these methods effectively reduce computational pressure by reducing the number of keys and values, the Swin Transformer limits the range of information that the model can directly capture, making it difficult for the model to capture long-distance dependencies beyond the window size; although the sparsity of PVT reduces the amount of computation, it also reduces the amount of information that the model can utilize, thereby affecting the modeling of long-distance relationships. They inevitably damage the model's ability to model long-distance relationships, and the damage to the modeling ability may directly affect the performance of the model; the localized attention mechanism limits the model's ability to generalize to specific data sets or tasks. When faced with new data with different feature distributions or more complex scenarios, the model may find it difficult to adapt and perform well.

[0006] In general, the fusion of multi-source data (i.e., multi-sensor data) in the autonomous driving field still faces the following key challenges: how to build a unified self-supervised learning framework to fully understand and exploit the characteristics of multi-source heterogeneous data and form effective multimodal representations, and how to design accurate and efficient multimodal feature fusion methods to promote robust environmental perception. These issues limit the performance of existing methods in complex scenarios. Therefore, an innovative solution that can achieve joint feature representation, accurate information alignment, and robust perception is urgently needed to promote the further development of autonomous driving perception technology. Summary of the Invention

[0007] The purpose of this application is to provide a self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving to solve the problems of insufficient understanding of the characteristics of multi-sensor data and poor environmental perception capabilities.

[0008] To achieve the above objectives, this application provides the following solutions:

[0009] This application provides a method for self-supervised pre-training and fusion perception of multi-sensor data for autonomous driving, including:

[0010] The multi-sensor data is processed to obtain voxelized point cloud data and image blocks, and the voxelized point cloud data and the image blocks are embedded as tags to obtain voxelized point cloud data tags and image data tags; the multi-sensor data includes lidar point cloud data and image data.

[0011] The voxelized point cloud data markers and the image data markers are masked to obtain image features and point cloud features, and a masked marker set is constructed.

[0012] The image features and point cloud features in the marker set are projected to obtain the positions of the image features and point cloud features on the camera view image.

[0013] Based on the multi-sensor data interaction module, the image features and point cloud features are fused according to their positions on the view image to obtain fused features.

[0014] The fused features are back-projected, and the scene is reconstructed based on the image feature branch and the point cloud feature branch to obtain a new scene.

[0015] Based on the new scene, the global information of the new scene is bridged through proxy tags to achieve fusion of multi-sensor data.

[0016] According to the specific embodiments provided in this application, this application has the following technical effects:

[0017] This application provides a self-supervised pre-training and fusion perception method for multi-sensor data in autonomous driving. By processing image-radar multi-sensor data, a three-dimensional point cloud voxel representation and image blocks are formed, and a feature embedding vector is generated. The point cloud voxels and image features are randomly masked to simulate real-world sensor failure, external noise, signal loss, and other conditions, thereby enhancing the robustness of feature learning. Based on the Multi-sensor Interaction Module (MIM), a joint representation with rich semantic information is formed through projection transformation according to the relative position relationship between the image and point cloud feature space. The scene is reconstructed through the image and point cloud branches respectively, and the optimal image-point cloud fusion features are learned in a self-supervised manner. Finally, an image-point cloud feature fusion perception algorithm based on the proxy attention mechanism is designed. The proxy labeling is used as a bridge to fuse global contextual scene information, fully exploiting the complementary characteristics of multi-sensor data, and improving the heterogeneous information fusion effect and environmental perception performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 A flowchart of the automatic driving multi-sensor data self-supervised pre-training and fusion perception method provided in the embodiments of the present application.

[0020] Figure 2 A working schematic diagram of the automatic driving multi-sensor data self-supervised pre-training and fusion perception method provided in the embodiments of the present application.

[0021] Figure 3 A schematic diagram of three attention mechanisms provided in the embodiments of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0023] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0024] As shown in Figure 2 The present application discloses an automatic driving multi-sensor data self-supervised pre-training and fusion perception method, which aims to promote the interaction and sharing of different sensor features and realize efficient fusion of image semantic features and point cloud geometric information.

[0025] On the one hand, a multi-sensor data self-supervised pre-training framework is constructed to realize the extraction and learning of multi-source data universal representation: firstly, camera and radar data features are extracted respectively and projected into an enhanced three-dimensional volume space (Volume), which makes up for the problem of missing height information in the Bird’s-eye View (BEV) perspective, and a camera-radar feature self-supervised pre-training model based on mask mechanism is proposed; then a multi-sensor interaction module (MIM) is designed to promote efficient communication and sufficient fusion of different sensor data information.

[0026] On the other hand, the multi-sensor feature fusion perception method based on the proxy attention mechanism of this application uses the proxy intelligent body as an intermediate medium to enhance the correlation between the query and key-value features, alleviate the interference of irrelevant noise on the fusion process and calculation delay, and efficiently improve the accuracy of multi-source information fusion and the robustness of perception.

[0027] This application fully leverages the potential of masked self-supervised learning and significantly reduces the dependence on large amounts of annotated data. It efficiently represents image and point cloud data through a unified three-dimensional voxel space (Volume) representation, and uses a multi-sensor data interaction module (MIM) to achieve deep interaction and information enhancement of image-point cloud features. It also combines the proxy attention fusion perception method to effectively solve the problems of multi-sensor data alignment difficulties, modal information loss, and poor fusion effects existing in traditional methods, while greatly improving the accuracy and robustness of three-dimensional spatial target perception in autonomous driving scenarios.

[0028] like Figure 1 As shown, the embodiment of the present application provides a method for self-supervised pre-training and fusion perception of multi-sensor data for autonomous driving, specifically including:

[0029] S1: Processing multi-sensor data to obtain voxelized point cloud data and image blocks, and embedding the voxelized point cloud data and the image blocks as labels to obtain voxelized point cloud data labels and image data labels; the multi-sensor data includes lidar point cloud data and image data.

[0030] S2: Masking the voxelized point cloud data markers and the image data markers to obtain image features and point cloud features, and constructing a masked marker set.

[0031] S3: Project the image features and point cloud features in the marker set to obtain the positions of the image features and point cloud features on the camera view image.

[0032] S4: Based on the multi-sensor data interaction module, the image features and point cloud features are fused according to their positions on the view image to obtain fused features.

[0033] S5: Back-projecting the fused features, and reconstructing the scene based on the image feature branch and the point cloud feature branch to obtain a new scene.

[0034] S6: Based on the new scene, global information of the new scene is bridged through proxy tags to achieve fusion of multi-sensor data.

[0035] Further, in an exemplary embodiment, step S1 may be replaced by the following steps.

[0036] S101: Mapping the three-dimensional spatial point positions in the lidar point cloud data to a voxel grid to obtain voxelized point cloud data.

[0037] The input LiDAR point cloud data is voxelized. Each 3D spatial point in the point cloud is mapped to a discrete voxel grid according to its position, thereby generating a voxelized point cloud feature representation.

[0038] S102: Embed the voxelized point cloud data as a label to obtain the voxelized point cloud data label.

[0039] The voxelized point cloud data is then embedded as tokens for subsequent feature extraction. Meanwhile, the input image data is processed in blocks. The image data is divided into non-overlapping blocks of a fixed size, and a positional encoding is added to each block to embed it as a token.

[0040] S103: performing block processing on the image data to obtain non-overlapping image blocks of a preset size.

[0041] S104: adding position codes to the non-overlapping image blocks to obtain image data labels.

[0042] This position encoding preserves the relative position information of image patches in two-dimensional space, allowing subsequent models to accurately capture the spatial relationships of image patches.

[0043] Further, in an exemplary embodiment, step S2 may be replaced by the following steps.

[0044] S201: Masking the voxelized point cloud data labels and the image data labels using a random masking strategy to obtain image features and point cloud features, and constructing a masked label set.

[0045] After generating the image and point cloud markers, a random masking strategy is used to partially mask the markers of the two modalities. This masking mechanism constructs a masked marker set ( , ),in( , ) is the feature set of the masked image I and point cloud voxel V. The masking operation not only simulates data loss in the real environment, but also forces the model to learn to infer hidden information from the unmasked markers, thereby improving the model's self-supervised learning ability.

[0046] Further, in an exemplary embodiment, step S3 may be replaced by the following steps.

[0047] S301: Based on the vehicle coordinate system, map each voxel of the point cloud feature to a corresponding position in the 3D volume space to obtain a projection of the point cloud feature.

[0048] Project image and point cloud features into a unified 3D volume space to achieve geometric and semantic fusion and alignment. The projection is based on the ego-vehicle coordinate system, achieved by mapping each voxel to a corresponding position in 3D volume space. The spatial information of the voxelized point cloud is precisely embedded into the 3D volume space using the 3D geometric properties of the point cloud data, using the vehicle as the reference frame. This creates a spatially consistent point cloud feature representation. This projection method ensures that the geometric accuracy of the point cloud data is not affected by any distortion.

[0049] S302: Utilize formula , obtaining the projection of the image feature; the positions of the image feature and the point cloud feature on the camera view image include the projection of the point cloud feature and the projection of the image feature.

[0050] in, is the image feature; i represents the camera view image; j represents the 3D reference point; The total number of points queried for each 3D voxel; is the eigenvalue of the jth point in the i-th camera view; is a projection function used to obtain the position of the j reference point on the i-th view image, and p is the position of the 3D point p projected onto the image plane of the i-th view to obtain the position of the point on the image; During the projection process, 3D points are only associated with a specific view; A set of query points for each 3D voxel.

[0051] For image features , the projection is completed using a 2D-3D spatial cross attention mechanism. Each image tag is queried for its corresponding position in the 3D volume space , using the camera's intrinsic and extrinsic parameters to project the corresponding 3D points into the 2D view. During the projection process, the 3D points are only associated with a specific view, called the view 2D features of 2D view from view The positions of these projected 3D reference points are sampled in .

[0052] in, is a set of 3D volume query points, ∈ is the number of channels; H is the height of the voxel in three-dimensional space; W is the width of the voxel in three-dimensional space; Z is the depth of the voxel in three-dimensional space.

[0053] Further, in an exemplary embodiment, step S4 may be replaced by the following steps.

[0054] S401: Concatenate image features and point cloud features to form a query vector.

[0055] We further propose a multi-sensor data interaction module to promote the two 3D voxel features. and Considering the high-performance requirements of downstream tasks for high resolution and the limitation of token sequence size in the standard self-attention mechanism, deformable self-attention is used here for fusion calculation to reduce the computational requirements. Each module consists of deformable self-attention, feedforward network and normalization: and Concatenate by channel dimension to form query vector .

[0056] S402: Input the query vector into a multi-sensor data interaction module to perform feature interaction to obtain a multimodal interaction feature; the multimodal interaction feature is a fusion feature.

[0057] Input into MMIM for feature interaction to form multimodal interaction features The specific process is as follows:

[0058] .

[0059] in, quantity, is the total number of sampled keys. and Respectively represent In the attention head The offset and attention weight of each sampling point. The value range of is [0, 1], and Normalization. Finally, is divided into channel dimensions to obtain 3D voxel features of a specific modality ( ).

[0060] Further, in an exemplary embodiment, step S5 may be replaced by the following steps.

[0061] S501: For the laser radar branch, project the sampling position of each voxel of the mask of the position of the point cloud feature on the view image in the vehicle coordinate system back to the 3D volume space to obtain the corresponding 3D feature; the laser radar branch is the point cloud feature branch.

[0062] Based on consistent 3D voxel features, the fused features can be efficiently and concisely projected back into the 2D view and 3D volume spaces, respectively. For the LiDAR branch, the mask voxel sampling position in the vehicle coordinate system is projected back into the original space to obtain the corresponding 3D features. After fusion, these features contain rich image semantic information.

[0063] S502: Based on the laser radar branch, use Reconstruct point cloud pixels; where, is the chamfer loss function; is the Chamfer distance function; is the voxel decoder; is the projected voxel feature; is the masked LiDAR point cloud divided into voxels.

[0064] S503: For the camera branch, use a projection function to project the 3D voxel feature of the position of the image feature on the view image to a 2D plane feature; the camera branch is an image feature branch.

[0065] Using the projection function 3D voxel features Project to 2D View feature Corresponding coordinate position Projection function The definition is as follows:

[0066] .

[0067] in, is the position in the 3D voxel, and are the intrinsic parameter matrix and extrinsic parameter matrix of the camera respectively, where It is a 4×4 matrix space.

[0068] S504: Based on the image feature branch, use the formula Reconstruct image pixels; where, is the loss of image reconstruction, is the mean square error loss MSE, is the original image in pixel space; is an image decoder; is the projected image feature.

[0069] The scene reconstruction task needs to consider the image and point cloud branches separately. On the one hand, the image pixel reconstruction adopts the mean square error (MSE) loss As a supervisory signal. On the other hand, point cloud voxel reconstruction uses Chamfer distance to measure the difference between two point sets at different scales. Chamfer loss Expressed as:

[0070] .

[0071] in, is the Chamfer distance function; is the voxel decoder; is the scene voxel prediction result; cumulative voxel reconstruction loss Defined as:

[0072] .

[0073] S505: Obtain a new scene according to the point cloud pixels and the image pixels.

[0074] Furthermore, in an exemplary embodiment, Figure 3 As shown, step S6 can be replaced by the following steps.

[0075] S601: Use the formula to bridge the global information of the query vector and the key-value vector to achieve the fusion of multi-sensor data; is the Agent attention mechanism; A is the agent , for Dimensional matrix space; is the Softmax attention mechanism; Q is the query value of the query vector; K is the key value of the key-value vector; V is the value matrix.

[0076] Considering the heterogeneous characteristics of image and point cloud features, the current mainstream attention method is difficult to achieve accurate and efficient fusion effect: Softmax attention needs to calculate the similarity between all queries (Q) and key-value pairs (K / V), which is computationally expensive; while linear attention uses a mapping function Adjusting the order of QKV vectors can reduce computational complexity, but this approach limits the ability to express features. The expressions of the two attention mechanisms are as follows:

[0077] .

[0078] .

[0079] in, is Softmax; is the Linear attention mechanism; Q is the query value of the query vector; K is the key value of the key value vector; V is the value matrix; Represents the Softmax function.

[0080] Therefore, a multi-source data fusion perception method based on the agent attention mechanism is proposed here, which elegantly bridges the global information of Q and K / V vectors through a set of agent tokens.

[0081] The calculation process of agent attention includes two parts: agent aggregation and agent broadcast. Agent Aggregation; For Agent Broadcast. Specifically, first mark the agent Initially treated as a query token, and using and Aggregate global features ; followed by As keys and values, with the query matrix Attention is calculated again, and the proxy features obtained from the global are broadcast back to each query token and form the final output This approach avoids the direct calculation of and The image-point cloud feature fusion process uses point cloud features as queries (Q), image features as key-value vectors (K / V), and the fusion result of the previous frame as an agent token. This iteratively fuses specific information from different modalities and scene context in both temporal and spatial dimensions to form comprehensive and robust multi-sensor fusion features for downstream perception tasks.

[0082] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for self-supervised pre-training and fusion perception of multi-sensor data for autonomous driving, characterized by: The autonomous driving multi-sensor data self-supervised pre-training and fusion perception method includes: Processing the multi-sensor data to obtain voxelized point cloud data and image blocks, and embedding the voxelized point cloud data and the image blocks as labels to obtain voxelized point cloud data labels and image data labels; the multi-sensor data includes lidar point cloud data and image data; Masking the voxelized point cloud data markers and the image data markers to obtain image features and point cloud features, and constructing a masked marker set; Project the image features and point cloud features in the marker set to obtain the positions of the image features and point cloud features on the camera view image; Based on the multi-sensor data interaction module, the image features and point cloud features are fused according to their positions on the view image to obtain fused features; Back-projecting the fused features, and reconstructing the scene based on the image feature branch and the point cloud feature branch to obtain a new scene; Based on the new scenario, global information of the new scenario is bridged through proxy tags to achieve fusion of multi-sensor data; The multi-sensor data interaction module fuses the image features and the point cloud features according to their positions on the view image to obtain fused features, specifically including: Concatenate image features and point cloud features to form a query vector; Inputting the query vector into a multi-sensor data interaction module for feature interaction to obtain a multimodal interaction feature; the multimodal interaction feature is a fusion feature; The back-projecting of the fused features and scene reconstruction based on the image feature branch and the point cloud feature branch to obtain a new scene specifically includes: For the LiDAR branch, based on the sampling position of each voxel of the mask of the position of the point cloud feature on the view image in the vehicle coordinate system, the sampled position is projected back to the 3D volume space to obtain the corresponding 3D feature; the LiDAR branch is a point cloud feature branch; Based on the laser radar branch, using Reconstruct point cloud pixels; where, is the chamfer loss function; is the Chamfer distance function; is the voxel decoder; is the projected voxel feature; is the masked LiDAR point cloud divided into voxels; For the camera branch, a projection function is used to project the 3D voxel feature of the position of the image feature on the view image to a 2D plane feature; the camera branch is an image feature branch; the 3D voxel feature is a 3D feature; Based on the image feature branch, using the formula Reconstruct image pixels; where, is the loss of image reconstruction, is the mean square error loss MSE, is the original image in pixel space; is an image decoder; is the projected image feature; A new scene is obtained according to the point cloud pixels and the image pixels.

2. The method for self-supervised pre-training and fusion perception of multi-sensor data for autonomous driving according to claim 1, characterized in that: Processing the multi-sensor data to obtain voxelized point cloud data and image blocks, and embedding the voxelized point cloud data and the image blocks as labels to obtain voxelized point cloud data labels and image data labels, specifically including: According to the three-dimensional space point position in the lidar point cloud data, it is mapped to the voxel grid to obtain voxelized point cloud data; Embedding the voxelized point cloud data as a label to obtain the voxelized point cloud data label; Performing block processing on the image data to obtain non-overlapping image blocks of a preset size; Position codes are added to the non-overlapping image blocks to obtain image data labels.

3. The method for self-supervised pre-training and fusion perception of multi-sensor data for autonomous driving according to claim 1, characterized in that: Masking the voxelized point cloud data markers and the image data markers to obtain image features and point cloud features, and constructing a masked marker set, specifically comprising: The voxelized point cloud data labels and the image data labels are randomly masked respectively by using a random masking strategy to obtain image features and point cloud features, and construct a masked label set.

4. The method for self-supervised pre-training and fusion perception of multi-sensor data for autonomous driving according to claim 1, characterized in that: Project the image features and point cloud features in the marker set to obtain their positions on the camera view image, specifically including: Based on the vehicle coordinate system, each voxel of the point cloud feature is mapped to the corresponding position in the 3D volume space to obtain the projection of the point cloud feature; using the formula , get the projection of image features; in, is the image feature; i represents the camera view image; j represents the 3D reference point; The total number of points queried for each 3D voxel; is the eigenvalue of the jth point in the i-th camera view; is a projection function used to obtain the position of the j reference point on the i-th view image, and p is the position of the 3D point p projected onto the image plane of the i-th view to obtain the position of the point on the image; During projection, 3D points are associated only with a specific view. A set of query points for each 3D voxel.

5. The method for self-supervised pre-training and fusion perception of multi-sensor data for autonomous driving according to claim 1, characterized in that: Based on the new scenario, global information is bridged through proxy tags to achieve the fusion of multi-sensor data, specifically including: Using the formula The query vector and the global information of the key-value vector are bridged to achieve the fusion of multi-sensor data; is the Agent attention mechanism; A is the agent , for Dimensional matrix space; is the Softmax attention mechanism; Q is the query value of the query vector; K is the key value of the key-value vector; V is the value matrix.

Images