Multi-modal automatic driving image segmentation method and system based on SAM model

Abstract

The embodiment of the invention discloses a multi-modal automatic driving image segmentation method and system based on an SAM model, and the method comprises the steps: a construction step: constructing the SAM model which comprises an image encoder, a mask decoder, a prompt encoder, a modal encoder and a depth perception module; a training step: training the model by using cross entropy loss and Dice loss; and an identification step: inputting the multi-modal automatic driving image into the model, and outputting a segmentation result of each object in the automatic driving image. According to the invention, the lightweight modal encoder and the depth sensing module are introduced, feature coding of depth information or other modal information is realized, the depth sensing module injects the coded depth information into a backbone network of the SAM model, and alternate enhancement of depth features and RGB features is realized, so that multi-modal expansion of the SAM model is realized, and the robustness of the SAM model is improved. And the huge calculation cost caused by retraining a large model for each multi-modal task is avoided.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving image recognition technology, and in particular to a multimodal autonomous driving image segmentation method and system based on a SAM model. Background Art

[0002] Existing large-scale visual models such as SAM (Segment Anything, a basic model proposed by Meta AI dedicated to image segmentation, which can predict object masks based on the hints of specified objects) use ultra-large-scale data for model training and demonstrate powerful image segmentation capabilities. However, SAM only uses RGB images during training, so it can only be used for RGB image segmentation, which limits the practical application of SAM in multimodal segmentation tasks.

[0003] In existing technologies, efficient fine-tuning methods have been able to adapt large visual models such as SAM to the field of autonomous driving. For example, LoRA (Low-Rank Adaptation) technology introduces low-rank matrices for fine-tuning, allowing large models to perform well in specific tasks without retraining the entire model.

[0004] However, the application of these technologies is mainly focused on the processing of single-modal images, especially RGB images. In the field of autonomous driving, more and more multi-modal recognition solutions of lidar images + multiple camera images are used to achieve 3D and 4D perception. Although these existing methods can efficiently adapt SAM to the field of autonomous driving, they cannot be extended to handle multi-modal segmentation tasks. For example, although LoRA technology can efficiently train and adapt models, it is limited to fine-tuning RGB image data and lacks support for other modalities such as depth images (compared to RGB images, the value of each pixel in the depth image can be used to characterize the distance between a certain point in the scene and the camera). Therefore, the application of existing technologies in multi-modal autonomous driving image segmentation tasks still has great limitations and cannot meet the growing processing needs of multi-modal autonomous driving image data. Summary of the invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a multimodal autonomous driving image segmentation method and system based on the SAM model to meet the processing requirements of multimodal autonomous driving image data.

[0006] In order to solve the above technical problems, an embodiment of the present invention proposes a multimodal autonomous driving image segmentation method based on a SAM model, comprising: Construction steps: construct a SAM model for segmenting multimodal autonomous driving images, the model comprising an image encoder, a mask decoder, a prompt encoder, a modality encoder and a depth perception module, wherein the modality encoder encodes the input non-RGB modality autonomous driving image to obtain non-RGB features of the non-RGB modality autonomous driving image; the depth perception module interacts with the RGB features according to the obtained non-RGB features, completes the enhancement of the non-RGB features and inputs them into the image encoder; Training step: using cross entropy loss and Dice loss to train the SAM model to obtain a trained SAM model, wherein parameters of the image encoder are frozen during the training process to improve training efficiency and reduce computing resource requirements; Recognition step: input the multimodal autonomous driving image into the trained SAM model, and the SAM model outputs the segmentation results of each object in the autonomous driving image according to the click prompt or the object box prompt.

[0007] Accordingly, an embodiment of the present invention further provides a multimodal autonomous driving image segmentation system based on a SAM model, comprising: Construction unit: construct a SAM model for segmenting multimodal autonomous driving images, the model comprising an image encoder, a mask decoder, a prompt encoder, a modality encoder and a depth perception module, wherein the modality encoder encodes the input non-RGB modality autonomous driving image to obtain non-RGB features of the non-RGB modality autonomous driving image; the depth perception module interacts with the RGB features according to the obtained non-RGB features, completes the enhancement of the non-RGB features and inputs them to the image encoder; A training unit: training the SAM model using a cross entropy loss and a Dice loss to obtain a trained SAM model, wherein the training unit freezes parameters of the image encoder during the training process to improve training efficiency and reduce computing resource requirements; Recognition unit: The multimodal autonomous driving image is input into the trained SAM model, and the SAM model outputs the segmentation results of each object in the autonomous driving image according to the click prompt or the object box prompt.

[0008] The beneficial effects of the present invention are: (1) By integrating a lightweight depth perception module, the model can process multimodal autonomous driving image data including non-RGB images without changing the original SAM structure. This means that SAM, which was originally only used for RGB image segmentation, can now be extended to the segmentation of multimodal autonomous driving image data such as RGB-D images, greatly improving the recognition efficiency of autonomous driving images.

[0009] (2) The present invention achieves a significant improvement in the performance of the SAM model while ensuring computing efficiency and resource utilization. Compared with retraining the entire model or using a more complex network structure, the present invention not only saves computing resources, but also speeds up the training of the model, while ensuring the high efficiency of multi-modal autonomous driving image data processing.

[0010] (3) The present invention has achieved excellent performance in multimodal autonomous driving image segmentation tasks. Experimental results show that the enhanced SAM significantly improves the segmentation accuracy and robustness of multimodal autonomous driving image data, and can more accurately process image information of various modalities in complex autonomous driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of a multimodal autonomous driving image segmentation method based on a SAM model according to an embodiment of the present invention.

[0012] Figure 2 It is a schematic diagram of the network structure of the SAM model of an embodiment of the present invention. DETAILED DESCRIPTION

[0013] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present invention is further described in detail below in conjunction with the drawings and specific embodiments.

[0014] In the embodiments of the present invention, if there are directional indications (such as up, down, left, right, front, back, etc.), they are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0015] In addition, in the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0016] Please refer to Figure 1 The multimodal autonomous driving image segmentation method based on the SAM model in an embodiment of the present invention includes a construction step, a training step, and a recognition step.

[0017] Construction steps: Construct a SAM model for segmenting multimodal autonomous driving images. The model includes an image encoder, a mask decoder, a prompt encoder, a modality encoder, and a depth perception module. The modality encoder encodes the input non-RGB modality autonomous driving image to obtain non-RGB features of the non-RGB modality autonomous driving image. The RGB modality autonomous driving image is an autonomous driving image captured by a visual camera. Non-RGB modality autonomous driving images include autonomous driving images captured by thermal imaging cameras, lidar, etc. Non-RGB modalities include thermal imaging camera modalities or lidar modalities, etc. Lidar can provide accurate depth information in the environment in the form of 3D point clouds, and is relatively less affected by light and weather. The following takes the autonomous driving image captured by lidar as an example (i.e., depth image) for explanation. Please refer to Figure 2 , the modal encoder encodes the input depth image and obtains the depth features of the depth image.

[0018] The depth perception module interacts with the RGB features according to the acquired depth features, completes the enhancement of the depth features and inputs them into the image encoder. The image encoder is used to map the image to be segmented to the image feature space. The prompt encoder is responsible for mapping the input prompt to the feature space of the prompt. The present invention adds a lightweight modal encoder and a depth perception module to the structure of the image encoder, mask decoder, and prompt encoder of the existing SAM model, gives the SAM model depth perception capabilities, and efficiently expands the SAM model to the multimodal autonomous driving image segmentation task, and achieves good performance, thereby realizing the efficient application of SAM in multimodal segmentation tasks.

[0019] The overall network structure diagram of the SAM model of the present invention is as follows: Figure 2 As shown on the left, the structure of the depth perception module (DepthPerception Layer) is as follows Figure 2 As shown on the right, Figure 2 The upper and lower figures of the SAM model input on the left are examples of LiDAR point cloud images (i.e., depth images) and RGB images. The parameters of the SAM Image Encoder are frozen and do not participate in training, which can greatly reduce the number of parameters for model training.

[0020] The original SAM model includes an image encoder, a mask decoder, and a prompt encoder. The present invention is based on the original SAM model and adds two modules, the modality encoder and the depth perception module to achieve the multimodal expansion of SAM. The present invention reduces the amount of training parameters of the model, thereby maximizing the capacity of the SAM large model and avoiding the problem of overfitting when there is less training data. Therefore, in the modality encoder, the present invention uses a small number of convolutional layers for modality encoding, for example, using three convolutional layers. For the depth perception module, the present invention uses two cross-modal attention modules to realize the interaction between RGB features and depth features.

[0021] The encoding of autonomous driving images in non-RGB modality is performed by the modality encoder Completed. Assume that the input non-RGB mode autonomous driving image data is , its width and height are respectively Then the features of the non-RGB modality autonomous driving image are The acquisition method is as follows: .

[0022] The present invention uses the depth perception module to realize the interaction between depth features and RGB features. This process is realized through two cross-modal attention modules. Layer depth features and RGB features Taking the fusion process of as an example, the formula of the cross-modal attention process is:

[0023] Here represents cross-modal attention computation, Represents the non-RGB feature (i.e., deep feature) information that the RGB modality pays attention to, and the enhanced RGB feature It can be expressed as . Through the After the network layer .

[0024] So far, the enhancement from depth to RGB has been completed. Next, the enhancement of RGB to depth feature information is introduced. The process is as follows: .

[0025] Training step: Use cross entropy loss and Dice loss to train the SAM model to obtain a trained SAM model. Most parameters are frozen during the training process to improve training efficiency and reduce computing resource requirements.

[0026] In this work, the inventors use the cross entropy loss and Dice loss To train the segmentation network: ; ; in, and Represents the height and width of the image respectively, represents the segmentation label, Represents the segmentation result of the model. The total training loss is and The linear addition of .

[0027] Recognition step: Input the multimodal autonomous driving image into the trained SAM model, and output the segmentation results of each object in the autonomous driving image. The multimodal autonomous driving image includes RGB images and non-RGB images (such as LiDAR point cloud images). The segmentation results are input into the recognition model of the relevant autonomous driving image for subsequent recognition.

[0028] The present invention avoids the huge computational overhead of retraining a large model for each multimodal task. The present invention realizes feature encoding of depth information or image information of other modalities by introducing a lightweight modal encoder and a depth perception module. The depth perception module injects the encoded depth information into the backbone network of the SAM model to realize alternating enhancement of depth features and RGB features, thereby realizing multimodal expansion of the SAM model.

[0029] The multimodal autonomous driving image segmentation system based on the SAM model of an embodiment of the present invention includes a construction unit, a training unit, and a recognition unit.

[0030] Construction unit: construct a SAM model for segmenting multimodal autonomous driving images, wherein the model includes an image encoder, a mask decoder, a prompt encoder, a modality encoder and a depth perception module, wherein the modality encoder encodes the input non-RGB modality autonomous driving image to obtain the depth features of the non-RGB modality autonomous driving image; the depth perception module interacts with the RGB features according to the acquired depth features, completes the enhancement of the depth features and inputs them to the image encoder.

[0031] Training unit: Use cross entropy loss and Dice loss to train the SAM model to obtain a trained SAM model. The training unit freezes most parameters during the training process to improve training efficiency and reduce computing resource requirements.

[0032] Recognition unit: input the multimodal autonomous driving image into the trained SAM model, so that the SAM model outputs the segmentation results of each object in the autonomous driving image according to the click prompt or the object frame prompt.

[0033] As an implementation method, the depth perception module includes two cross-modal attention modules, which respectively enhance the depth feature to the RGB feature and enhance the depth feature from the RGB feature. The enhancement process formula of RGB features is: ; The formula for the enhancement process of deep features is: ; in, ; is the first The non-RGB features of the layer (i.e., depth features), is the first RGB features of the layer, Represents the deep feature information of the autonomous driving image in RGB mode, is the enhanced RGB feature, Represents cross-modal attention computation.

[0034] As an implementation mode, the training unit trains the SAM model using the following formula: ; ; in, and Represents the height and width of the image respectively, represents the segmentation label, represents the segmentation result of the SAM model, is the cross entropy loss, Dice loss.

[0035] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal autonomous driving image segmentation method based on the SAM model, characterized in that: include: Construction steps: construct a SAM model for segmenting multimodal autonomous driving images, the model comprising an image encoder, a mask decoder, a prompt encoder, a modality encoder and a depth perception module, wherein the modality encoder encodes the input non-RGB modality autonomous driving image to obtain non-RGB features of the non-RGB modality autonomous driving image; the depth perception module interacts with the RGB features according to the obtained non-RGB features, completes the enhancement of the non-RGB features and inputs them into the image encoder; Training step: using cross entropy loss and Dice loss to train the SAM model to obtain a trained SAM model, wherein parameters of the image encoder are frozen during the training process to improve training efficiency and reduce computing resource requirements; Recognition step: input the multimodal autonomous driving image into the trained SAM model, and the SAM model outputs the segmentation results of each object in the autonomous driving image according to the click prompt or the object box prompt.

2. The multimodal autonomous driving image segmentation method based on the SAM model according to claim 1, characterized in that: The depth perception module includes two cross-modal attention modules, which respectively enhance the non-RGB features to RGB features and enhance the RGB features to non-RGB features. The enhancement process formula of RGB features is: ； The enhancement process formula of non-RGB features is: ； in, ; is the first Non-RGB features of the layer, is the first RGB features of the layer, Represents the non-RGB feature information that autonomous driving images in RGB mode pay attention to, is the enhanced RGB feature, Represents cross-modal attention computation.

3. The multimodal autonomous driving image segmentation method based on the SAM model as claimed in claim 1, characterized in that: In the training step, the SAM model is trained using the following formula: ； ； in, and Represents the height and width of the image respectively, represents the segmentation label, represents the segmentation result of the SAM model, is the cross entropy loss, Dice loss.

4. A multimodal autonomous driving image segmentation system based on the SAM model, characterized in that: include: Construction unit: construct a SAM model for segmenting multimodal autonomous driving images, the model comprising an image encoder, a mask decoder, a prompt encoder, a modality encoder and a depth perception module, wherein the modality encoder encodes the input non-RGB modality autonomous driving image to obtain non-RGB features of the non-RGB modality autonomous driving image; the depth perception module interacts with the RGB features according to the obtained non-RGB features, completes the enhancement of the non-RGB features and inputs them to the image encoder; A training unit: training the SAM model using a cross entropy loss and a Dice loss to obtain a trained SAM model, wherein the training unit freezes parameters of the image encoder during the training process to improve training efficiency and reduce computing resource requirements; Recognition unit: The multimodal autonomous driving image is input into the trained SAM model, and the SAM model outputs the segmentation results of each object in the autonomous driving image according to the click prompt or the object box prompt.

5. The multimodal autonomous driving image segmentation system based on the SAM model as claimed in claim 4, characterized in that: The depth perception module includes two cross-modal attention modules, which respectively enhance the non-RGB features to RGB features and enhance the RGB features to non-RGB features. The enhancement process formula of RGB features is: ； The enhancement process formula of non-RGB features is: ； in, ; is the first Non-RGB features of the layer, is the first The RGB features of the layer, Represents the non-RGB feature information that autonomous driving images in RGB mode pay attention to, is the enhanced RGB feature, Represents cross-modal attention computation.

6. The multimodal autonomous driving image segmentation system based on the SAM model as claimed in claim 4, characterized in that: The training unit trains the SAM model using the following formula: ； ； in, and Represents the height and width of the image respectively, represents the segmentation label, represents the segmentation result of the SAM model, is the cross entropy loss, For Dice loss.

Images