Multimodal self-supervised pre-training method for autonomous driving based on bird's-eye view contrastive learning

Abstract

The present invention discloses a multimodal self-supervised pre-training method for autonomous driving based on bird's-eye view contrastive learning, which belongs to the field of computer vision technology. This method uses different data augmentations to transform LiDAR point cloud data and image data. After the network extracts features respectively, the two modal features are projected into the same bird's-eye view (BEV) space. The features of the two are then aligned using inverse transformation, and the network is enabled to learn general LiDAR point cloud and image representations through fine-grained contrastive learning. In practical applications, the trained network with strong representation capabilities can be applied to a variety of downstream tasks of autonomous driving, including but not limited to 3D target detection, 3D instance segmentation, semantic segmentation and other tasks. The above-mentioned pre-trained network can be used to initialize the backbone network of the detector, accelerate the convergence speed of the model during training, and improve network performance.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and involves pre-training an autonomous driving neural network model using computer vision and deep learning technology. In particular, it relates to a multimodal self-supervised pre-training method based on bird's-eye view comparative learning, which can be used for a general neural network model for autonomous driving. Background Art

[0002] In recent years, machine learning research and applications have flourished. Deep learning methods have achieved critical progress in many application areas, such as computer vision, speech processing, and machine translation. The success of deep learning in perceptual tasks is largely attributed to the powerful image representation and learning capabilities of convolutional neural networks. This powerful representation and learning capability enables the network to transfer between different tasks. Therefore, in practical applications, neural network training is generally divided into two steps. First, the network is pre-trained on a large-scale classification task dataset, allowing the network to learn powerful visual representations suitable for image understanding tasks. Then, the representation capabilities learned by the network on the large-scale dataset are transferred to downstream tasks with relatively smaller datasets, such as object detection, semantic segmentation, and instance segmentation. This combined pre-training and fine-tuning training paradigm can achieve better results than directly training on the downstream task.

[0003] However, pre-training networks for large-scale classification tasks relies on large amounts of manually labeled data. Even with the availability of large-scale classification datasets like ImageNet, ImageNet is increasingly unable to meet the training requirements as network architectures become more complex and larger. However, acquiring larger labeled datasets is extremely time-consuming and labor-intensive. Therefore, to reduce the workload of data labeling, researchers have focused on how to effectively utilize large amounts of unlabeled data. Consequently, self-supervised learning, which learns visual representations of images without manual annotation, has emerged.

[0004] Self-supervised learning, a method of unsupervised learning, aims to automatically generate pseudo-labels for unlabeled data through designed self-supervised tasks, without the need for image annotation. These pseudo-labels and the corresponding self-supervised tasks are then used to pre-train the neural network. Image representations obtained through self-supervised pre-training have better generalization capabilities than supervised pre-training and often achieve better results when transferred to downstream tasks.

[0005] In practical applications, existing self-supervised pre-training methods typically employ contrastive learning for self-supervised tasks. For downstream tasks in autonomous driving, such as 3D object detection, high-precision network architectures typically have separate processing components for LiDAR point clouds and image data. While contrastive learning-based self-supervised pre-training methods can pre-train LiDAR and image networks separately, these methods often impose strong constraints on the LiDAR network architecture and lack interaction between the two modalities, resulting in low pre-training efficiency.

[0006] Therefore, the existing self-supervised pre-training method based on contrastive learning still has certain technical bottlenecks for downstream tasks in autonomous driving, the network constraints are too large, and the pre-training efficiency is low. Summary of the Invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a multimodal self-supervised pre-training method based on bird's-eye view contrastive learning for autonomous driving. It proposes to use different data augmentations to transform LiDAR point cloud data and image data. After the network extracts features respectively, the two modal features are projected into the same bird's-eye view (BEV) space. Then, the features of the two are aligned using inverse transformation. Through fine-grained contrastive learning, the network can learn universal LiDAR point cloud and image representations, thereby improving the performance of downstream tasks of autonomous driving (such as 3D object detection, 3D instance segmentation, point cloud semantic segmentation, etc.).

[0008] The technical solution of the present invention is:

[0009] A multimodal self-supervised pre-training method for autonomous driving includes the following steps:

[0010] 1) Based on the required task, select two backbone networks for LiDAR point cloud input and image input, and add a depth predictor to the image backbone network;

[0011] 2) Build a multimodal data augmentation strategy. For LiDAR input, we use rotation, translation, and symmetry data augmentation strategies. For image input, we use cropping, color transformation, and symmetry data augmentation strategies.

[0012] 3) Input the autonomous driving data (including LiDAR and images) into the multimodal data augmentation strategy in step 2) to obtain the augmented data, which is then input into the backbone network of the corresponding modality to obtain the feature representation of the corresponding modality. The feature representation of the image is input into the depth predictor to obtain a depth map representing the image in pixel space.

[0013] 4) Construct a multimodal bird's-eye view feature. Using the depth map obtained in step 3), the 2D image representation is converted to a 3D image representation. The 3D image representation is then converted to the LiDAR space using the transformation matrix between LiDAR space and pixel space. A compression operation is then performed to convert the converted 3D image representation and the original LiDAR representation into a bird's-eye view representation.

[0014] 5) Multimodal bird's-eye view feature alignment: the bird's-eye view features of the two modalities are transformed using inverse data transformation to obtain aligned bird's-eye view features;

[0015] 6) Multimodal fine-grained contrastive learning: The cosine similarity and absolute distance of the bird's-eye view features of the two modalities are used as weights. The Hungarian algorithm is used to match the bird's-eye view representations of the two modalities. The matched representations are used as positive samples, and the non-matching representations are used as negative samples for contrastive learning to achieve multimodal self-supervised pre-training for autonomous driving.

[0016] Furthermore, after pre-training, for different downstream tasks (such as 3D object detection, 3D instance segmentation, and point cloud semantic segmentation, etc.), the pre-trained weights are loaded into the backbone network part of the downstream task network, and fine-tuned according to the training process of the downstream task to obtain the final neural network model related to the downstream task.

[0017] The above pre-training algorithm can be widely used in computer vision tasks (such as 3D object detection, 3D instance segmentation, and point cloud semantic segmentation) used in practical applications of autonomous driving to improve the performance of tasks (such as detection and segmentation accuracy).

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The present invention provides a multimodal self-supervised pre-training method based on bird's-eye view contrastive learning for autonomous driving, and proposes to transform LiDAR point cloud data and image data using different data augmentations. After the network extracts features respectively, the two modal features are projected into the same bird's-eye view (BEV) space, and then the features of the two are aligned using inverse transformation. Through fine-grained contrastive learning, the network is enabled to learn universal LiDAR point cloud and image representations, thereby improving the performance of downstream tasks of autonomous driving (such as 3D target detection, 3D instance segmentation, point cloud semantic segmentation, etc.). The method of the present invention breaks the conventional single-modal pre-training idea for autonomous driving data sets, establishes a multimodal joint pre-training method based on bird's-eye view contrastive learning, and realizes efficient and effective pre-training. The pre-training method of the present invention is universal, does not rely on a specific network architecture, is applicable to all current network models for autonomous driving data sets, and can be applied to a variety of computer vision tasks (3D target detection, 3D instance segmentation, point cloud semantic segmentation, etc.). The method is simple and convenient, can effectively improve the accuracy, and accelerate the convergence speed of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the pre-training method proposed in the present invention.

[0021] Figure 2 This is a flowchart of the application of the pre-training model provided by the present invention to downstream tasks. DETAILED DESCRIPTION

[0022] The present invention proposes a multimodal self-supervised pre-training method for autonomous driving based on bird's-eye view comparative learning. The present invention is further described below through examples in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0023] The specific embodiment of the present invention comprises the following steps (such as Figure 1 shown):

[0024] 1) Build a LiDAR backbone network and select the corresponding LiDAR backbone network based on the required downstream tasks, such as PointPillar, SECOND, etc.

[0025] 2) Build a multi-view image backbone network. Based on the required downstream tasks, select the corresponding image backbone network, such as ResNet, SwinTransformer, etc. At the same time, build a depth predictor composed of a single linear layer.

[0026] 3) Construct a multimodal data augmentation strategy. For LiDAR input, a rotation, translation and symmetric data augmentation strategy is adopted, denoted as R L, for image input, cropping, color transformation and symmetrical data augmentation strategies are adopted, denoted as R I .

[0027] 4) Input the autonomous driving data (including LiDAR and images) into the multimodal data augmentation strategy in step (3) to obtain the augmented data, which is then input into the backbone network of the corresponding modality. The output of the LiDAR backbone network is denoted as F L , the output of the image backbone network is recorded as F I At this time, F L is the voxel feature, F I is a 2D pixel feature.

[0028] 5) Construct multimodal bird's-eye view features and transform F I Input into the depth predictor to get the depth of each pixel in the pixel space. Using this depth, F I Convert 2D pixel features to 3D features. Then, according to the camera internal and external parameters, F I Convert from pixel space to LiDAR space. L With F I The coordinates of F are already in the same LiDAR space. L and F I All are compressed along the z-axis and converted into a 2D bird's-eye view feature F LBEV and F IBEV .

[0029] 6) Align the multimodal bird’s-eye view features and align the bird’s-eye view features F LBEV and F IBeV Use inverse data transformation and Transform to get aligned bird's-eye view features

[0030] 7) Multimodal fine-grained contrastive learning, which combines the cosine similarity of the bird's-eye view features of the two modalities with the normalized absolute distance

[0031]

[0032] As weights, the Hungarian algorithm is used to align the bird's-eye view features. and In the example, a global match M=assign(cost=w) is performed on each position, and the matching pairs are recorded as The matching pairs are taken as positive samples, the non-matching pairs as negative samples, and InfoNCE is used as the loss function:

[0033]

[0034] To update the network for pre-training, where v i Features The eigenvectors in Features The eigenvectors in It is expressed as the vector dot product of the two, where τ is the temperature coefficient and is set to 0.5.

[0035] 8) After pre-training, if Figure 2 As shown in the figure, for different downstream tasks (such as 3D object detection, 3D instance segmentation, and point cloud semantic segmentation), the pre-trained weights are loaded into the backbone network part of the downstream task network, and fine-tuned according to the training process of the downstream task to obtain the final neural network model related to the downstream task.

[0036] The above pre-training algorithm can be widely used in computer vision tasks (such as 3D object detection, 3D instance segmentation, and point cloud semantic segmentation) used in practical applications of autonomous driving to improve the performance of tasks (such as detection and segmentation accuracy).

[0037] The multimodal self-supervised pre-training method for autonomous driving provided by the present invention enables the network to learn universal LiDAR point cloud and image representations, achieves efficient and effective pre-training, and improves the performance of downstream tasks of autonomous driving (such as 3D object detection, 3D instance segmentation, point cloud semantic segmentation, etc.).

[0038] In the 3D object detection task, the method of the present invention was specifically implemented on the object detection dataset nuScenes. The results of the implementation on nuScenes show that, taking the CenterPoint detector as an example, under the same training conditions, using the method of the present invention for pre-training can improve the CenterPoint detector's mAP value on the validation set val from 56.03 to 57.25. (Note: nuScenes is a large-scale autonomous driving dataset that includes tasks such as 3D object detection, 2D object detection, and segmentation. The mAP value is a metric for measuring detection performance.)

[0039] It should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A multimodal self-supervised pre-training method for autonomous driving, characterized in that: The steps include: 1) Based on the required task, two backbone networks are selected for LiDAR point cloud input and image input, and a depth predictor is added to the image backbone network; 2) Constructing a multimodal data augmentation strategy; 3) Inputting the autonomous driving data into the multimodal data augmentation strategy in step 2) to obtain augmented data, then inputting this data into the backbone network of the corresponding modality to obtain the feature representation of the corresponding modality, and inputting the feature representation of the image into the depth predictor to obtain a depth map of the image representation in pixel space; 4) Construct a multimodal bird's-eye view feature. Use the depth map obtained in step 3) to convert the 2D image representation into a 3D image representation. Simultaneously, use the conversion matrix between LiDAR space and pixel space to convert the 3D image representation to LiDAR space. Then, perform a compression operation to convert the converted 3D image representation and the original LiDAR representation into a bird's-eye view representation. 5) Multimodal bird's-eye view feature alignment: the bird's-eye view features of the two modalities are transformed using inverse data transformation to obtain aligned bird's-eye view features; 6) Multimodal fine-grained contrastive learning: The cosine similarity and absolute distance of the bird's-eye view features of the two modalities are used as weights. The Hungarian algorithm is used to match the bird's-eye view representations of the two modalities. The matched representations are used as positive samples, and the non-matching representations are used as negative samples for contrastive learning to achieve multimodal self-supervised pre-training for autonomous driving.

2. The multimodal self-supervised pre-training method for autonomous driving according to claim 1, wherein: In step 6), InfoNCE is used as the loss function: To update the network for pre-training, where v i Characterized by The eigenvectors in Characterized by The eigenvectors in It is expressed as the vector dot product of the two, and τ is the temperature coefficient.

3. The multimodal self-supervised pre-training method for autonomous driving according to claim 1, wherein: In step 2), for the LiDAR input, rotation, translation and symmetric data augmentation strategies are adopted.

4. The multimodal self-supervised pre-training method for autonomous driving according to claim 1, wherein: In step 2), cropping, color transformation and symmetrical data augmentation strategies are adopted for image input.

5. The multimodal self-supervised pre-training method for autonomous driving according to claim 1, wherein: Step 6) After pre-training, for different downstream tasks, the pre-trained weights are loaded into the backbone network part of the downstream task network, and fine-tuned according to the training process of the downstream task to obtain the neural network model related to the downstream task.

Images