3D self-supervised pre-training method based on large-scale language-image model guidance

Abstract

This invention discloses a 3D self-supervised pre-training method based on a large-scale language-image model, belonging to the field of computer vision technology. This invention uses the semantic understanding and multimodal processing capabilities of the large-scale language-image model as guiding objectives, aligning the 3D network features with the feature space of the large-scale language-image model using contrastive learning. Furthermore, by utilizing image masking, 3D features and language descriptions are used to reconstruct the masked image, enabling the 3D network to learn the same universal LiDAR point cloud representation as the large-scale language-image model. In practical applications, the trained network with strong representational capabilities can be applied to various downstream tasks of autonomous driving, including but not limited to 3D object detection, 3D instance segmentation, and semantic segmentation, accelerating the convergence speed of the model during downstream task training and improving network performance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to the pre-training of autonomous driving neural network models using computer vision and deep learning technologies. In particular, it relates to a 3D self-supervised pre-training method based on a large-scale language-image model, which can be used for general neural network models for autonomous driving. Background Technology

[0002] In recent years, machine learning research and applications have flourished. Deep learning methods, in particular, have made crucial progress in many application areas, such as computer vision, speech processing, and machine translation. Autonomous driving technology has also demonstrated enormous potential in the automotive industry alongside the development of deep learning. Achieving fully autonomous driving systems requires solving many complex problems, including perception, decision-making, and control. Perception is the core technology of autonomous driving systems; it acquires environmental information from various sensors and accurately understands and infers the position, speed, and potential future behavior of roads, obstacles, and other vehicles, providing crucial information for subsequent driving planning.

[0003] In the training of perception tasks for autonomous driving, obtaining large-scale, high-quality datasets is a prerequisite for model training. However, manual data annotation in 3D scenes is an expensive and time-consuming task. Therefore, it is of great significance to implement a method that can utilize large-scale unlabeled data to assist in the training of perception tasks for autonomous driving.

[0004] Self-supervised training is an effective method for training using large-scale unlabeled data. However, existing self-supervised pre-training methods for 3D autonomous driving scenarios typically only target LiDAR point clouds and lack interaction between multiple modalities, resulting in low pre-training efficiency. Therefore, existing 3D self-supervised pre-training methods still face certain technical bottlenecks and low pre-training efficiency for autonomous driving scenarios.

[0005] In recent years, large-scale language-image models have achieved significant breakthroughs in the field of natural language processing. These models, through a large number of unlabeled text-image pairs, can automatically learn the semantic information of images and text, possessing powerful language and image modeling and generation capabilities. However, existing large-scale language-image models can only jointly model language and images and cannot directly process LiDAR point clouds. Although some existing methods can introduce large-scale language-image models into indoor point clouds, the data characteristics of LiDAR point clouds in autonomous driving scenarios differ significantly from those of indoor point clouds, making direct transfer between the two methods a considerable challenge. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a 3D self-supervised pre-training method guided by a large-scale language-image model to alleviate the problems of data scarcity and labeling difficulties in training autonomous driving perception models. This method uses the semantic understanding and multimodal processing capabilities of a large-scale language-image model as guiding objectives. By aligning image-LiDAR point cloud pairs and employing contrastive learning and mask learning, textual information is introduced into the 3D perception model. This enables the 3D perception model to extract rich features and representations from unlabeled autonomous driving data, improving the performance of downstream autonomous driving tasks (such as 3D object detection, 3D instance segmentation, and point cloud semantic segmentation).

[0007] The technical solution of this invention is:

[0008] A 3D self-supervised pre-training method based on a large-scale language-image model includes the following steps:

[0009] 1) Select the backbone network and large-scale language-image model for LiDAR point cloud input based on the required task;

[0010] 2) Construct a multimodal data augmentation strategy;

[0011] 3) Construct LiDAR-image-text pairs, input the raw image data of autonomous driving into a large-scale language-image model, and obtain multiple language descriptions of the images;

[0012] 4) Input the autonomous driving data into the multimodal data augmentation strategy in step 2) to obtain augmented data. Input the LiDAR data into the point cloud backbone network in step 1), input the image data into the image encoder of the large-scale language-image model, and input the text data into the text encoder of the large-scale language-image model. The output of the LiDAR backbone network is denoted as F. L The output of the image encoder is denoted as F. I The output of the text encoder is denoted as F. T

[0013] 5) Randomly mask the image, and input the masked image into the image encoder of the large-scale language-image model to obtain the feature representation of the masked image.

[0014] 6) Multimodal feature contrastive learning: Features from aligned text and image modalities are used as positive samples, and unaligned features are used as negative samples to contrast and learn LiDAR features.

[0015] 7) Combine LiDAR and text features F L and F TAs a cue word or enhancement feature, it is used in conjunction with the mask image features in step 5). Together, they are input into the image decoder of the generative model to reconstruct the mask content I of the image. m .

[0016] Furthermore, InfoNCE is used as the loss function to perform comparative learning on LiDAR features.

[0017]

[0018] Where τ is the temperature coefficient. and Let represent the output features of the LiDAR backbone network, the output image features of the image encoder, and the output text features of the text encoder for the i-th sample during training, respectively. and Let represent the output features of the LiDAR backbone network, the output image features of the image encoder, and the output text features of the text encoder for the j-th sample during training, respectively.

[0019] Mask content I of the reconstructed image m Calculate the L2 loss function with the unmasked image I:

[0020]

[0021] Where I m The image is a graphical mask for network reconstruction, where I represents the content of the real image. This indicates that the summation is performed using the L2 norm, and the loss function is... The pre-trained weights are merged and used as the final training loss function, and gradient directional propagation is used to update the pre-trained weights.

[0022] Furthermore, after pre-training, for different downstream tasks (such as 3D object detection, 3D instance segmentation, and point cloud semantic segmentation), the pre-trained weights are loaded into the backbone network of the downstream task network and fine-tuned according to the training process of the downstream task to obtain the final neural network model related to the downstream task.

[0023] This invention utilizes aligned point cloud and image data to align 3D network features with the feature space of a large-scale language-image model using contrastive learning. Furthermore, it employs image masking, using 3D features and language descriptions to reconstruct the masked image, enabling the 3D network to learn the same universal LiDAR point cloud representation as the large-scale language-image model. In practical applications, the trained network with strong representational capabilities can be applied to various downstream tasks in autonomous driving, including but not limited to 3D object detection, 3D instance segmentation, and semantic segmentation. The pre-trained network can be used to initialize the backbone network, accelerating the convergence speed of the model during downstream task training and improving network performance.

[0024] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention breaks away from the conventional single-modal pre-training approach for autonomous driving datasets, establishing a multi-modal joint pre-training method guided by a large-scale language-image model, achieving efficient and effective pre-training. The pre-training method of this invention is universal, independent of specific network architectures, applicable to all current network models for autonomous driving datasets, and can be applied to various computer vision tasks (3D object detection, 3D instance segmentation, point cloud semantic segmentation, etc.). The method is simple and convenient, effectively improving accuracy and accelerating model convergence. Attached Figure Description

[0025] Figure 1 This is a flowchart of a 3D self-supervised pre-training method according to a specific embodiment of the present invention.

[0026] Figure 2 This is a flowchart of a multimodal contrastive learning method according to a specific embodiment of the present invention.

[0027] Figure 3 This is a flowchart of a multimodal mask prediction method according to a specific embodiment of the present invention.

[0028] Figure 4 This is a flowchart of the application of the pre-trained model provided by this invention to downstream tasks. Detailed Implementation

[0029] This invention proposes a 3D self-supervised pre-training method based on a large-scale language-image model. The invention is further described below with reference to the accompanying drawings and through embodiments, but this does not limit the scope of the invention in any way.

[0030] The specific embodiments of the present invention include the following steps (e.g. Figure 1 As shown):

[0031] 1) Construct a LiDAR backbone network. Based on the required downstream tasks, select the corresponding LiDAR backbone network, such as PointPillar and SparseConv.

[0032] 2) Construct large-scale language-image models. Based on the required downstream tasks, select the corresponding language-image models, including question-answering models and generative models, such as LLaMA and StableDiffusion.

[0033] 3) Construct a multimodal data augmentation strategy. For LiDAR input, a rotation, translation, and symmetry data augmentation strategy is adopted, denoted as R. L For image input, a data augmentation strategy involving cropping, color transformation, and symmetry is employed, denoted as R. I .

[0034] 4) Construct LiDAR-image-text pairs, input the raw image data of autonomous driving into a large-scale language-image model, and obtain multiple language descriptions of the images.

[0035] 5) Input the autonomous driving data (including LiDAR and images) into the multimodal data augmentation strategy in step (3) to obtain augmented data. Input the LiDAR, image, and text pairs into the corresponding modal backbone network or encoder. The output of the LiDAR backbone network is denoted as F. L The output of the image encoder is denoted as F. I The output of the text encoder is denoted as F. T .

[0036] 6) Randomly mask the image, and then input the masked image into the image encoder of the large-scale language-image model to obtain the feature representation of the masked image.

[0037] 7) such as Figure 2 As shown, multimodal feature contrastive learning uses aligned text and image features as positive samples and unaligned features as negative samples, employing InfoNCE as the loss function to perform contrastive learning on LiDAR features.

[0038]

[0039] Where τ is the temperature coefficient, set to 0.5. and Let represent the output features of the LiDAR backbone network, the output image features of the image encoder, and the output text features of the text encoder for the i-th sample during training, respectively. and Let represent the output features of the LiDAR backbone network, the output image features of the image encoder, and the output text features of the text encoder for the j-th sample during training, respectively.

[0040] 8) For example Figure 3 As shown, LiDAR and text features F L and F T As a cue word or enhancement feature, it is related to the mask image features in (6). Together, they are input into the image decoder of the generative model to reconstruct the mask content I of the image. m Calculate the L2 loss function with the unmasked image I:

[0041]

[0042] Where I m The image is a graphical mask for network reconstruction, where I represents the content of the real image. This indicates that the summation is performed using the L2 norm. The loss function is... The weights are merged and used as the final training loss function, and the pre-trained weights are updated using gradient directional propagation, while the weights of the large-scale language-image model remain unchanged.

[0043] 9) After pre-training, if Figure 4 As shown, for different downstream tasks (such as 3D object detection, 3D instance segmentation, and point cloud semantic segmentation), the pre-trained weights are loaded into the backbone network of the downstream task network and fine-tuned according to the training process of the downstream task to obtain the final neural network model related to the downstream task.

[0044] The 3D self-supervised pre-training method based on a large-scale language-image model provided by this invention enables the network to learn general LiDAR point cloud and image representations, achieving efficient and effective pre-training and improving the performance of downstream tasks in autonomous driving. It can be widely applied to computer vision tasks in practical autonomous driving applications (such as 3D object detection, 3D instance segmentation, and point cloud semantic segmentation) to improve task performance (such as detection and segmentation accuracy).

[0045] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A 3D self-supervised pre-training method based on large-scale language-image model guidance, comprising the following steps: 1) Selecting a backbone network for LiDAR point cloud input and a large-scale language-image model according to the required task; 2) Building a multi-modal data augmentation strategy; 3) Building a LiDAR-image-text pair, inputting the original image data of autonomous driving into the large-scale language-image model to obtain multiple language descriptions of the image; 4) input the autonomous driving data into the multi-modal data augmentation strategy in step 2) to obtain augmented data, input the LiDAR data into the point cloud backbone network in step 1), input the image data into the image encoder of the large-scale language-image model, and input the text data into the text encoder of the large-scale language-image model, the output of the LiDAR backbone network is denoted as , the output of the image encoder is denoted as , and the output of the text encoder is denoted as ; 5) randomly mask the image, input the masked image into the image encoder of the large-scale language-image model to obtain the feature representation of the masked image ; 6) Multi-modal feature contrast learning, taking the features of the aligned text and image two modalities as positive samples and the non-aligned features as negative samples to perform contrast learning on the LiDAR features; 7) input LiDAR features and textual features as prompt words or augmentation features together with the masked image features in step 5) into the image decoder of the generative model to reconstruct the masked content of the image .​ 2. The 3D self-supervised pre-training method based on large-scale language-image model guidance according to claim 1, wherein, In step 2), for LiDAR input, a rotation, translation and symmetry data augmentation strategy is adopted.

3. The 3D self-supervised pre-training method based on large-scale language-image model guidance according to claim 1, wherein, In step 2), for image input, a cropping, color transformation and symmetry data augmentation strategy is adopted.

4. The 3D self-supervised pre-training method based on large-scale language-image model guidance according to claim 1, wherein, In step 6), InfoNCE is used as the loss function to perform contrast learning on the LiDAR features in For temperature coefficient, , and They represent the training time. Output features of the LiDAR backbone network, output image features of the image encoder, and output text features of the text encoder for each sample. , and They represent the training time. Output features of the LiDAR backbone network, output image features of the image encoder, and output text features of the text encoder for each sample.

5. The 3D self-supervised pre-training method based on large-scale language-image model guidance according to claim 4, wherein, Mask content of the reconstructed image in step 7) Non-masked image Compute L2 loss function: wherein is a graph mask content of the network reconstruction, represents a content of a real image, represents a summing thereof using a two-norm, the loss function being merged as a final training loss function, the pre-trained weights being updated with gradient direction propagation.

6. The 3D self-supervised pre-training method based on large-scale language-image model guidance according to claim 5, wherein, After pre-training, for different downstream tasks, the pre-trained weights are loaded into the backbone network part of the downstream task network to obtain the final neural network model related to the downstream task.

Images