A segmentation method for multi-modal weak-texture slope remote sensing images based on SAM

Abstract

The application discloses a kind of based on SAM's multi-modal weak texture side slope remote sensing image segmentation method, the method is as follows: step 1, construct weak texture side slope segmentation model based on SAM network;Step 2, to multi-modal weak texture side slope remote sensing image is carried out bilateral filtering and normalization processing;Step 3, combination uses Dice loss and cross-entropy loss training weak texture side slope segmentation model, and the parameter of image encoder is frozen in training process;Step 4, the multi-modal weak texture side slope remote sensing image after processing is input to trained weak texture side slope segmentation model, and the weak texture side slope segmentation model based on SAM network outputs the segmentation result of side slope fissure feature in multi-modal weak texture side slope remote sensing image.The method introduces lightweight HRSI module and modal perception module, and introduces multi-scale attention mechanism in image encoder, realizes the efficient application of SAM in multi-modal weak texture side slope image segmentation task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of surveying and mapping, and relates to a texture feature segmentation method, in particular to a texture feature segmentation method applied to multi-modal weak texture slope remote sensing images. BACKGROUND

[0002] At present, the main means for quickly obtaining key information of rock mass joints, fissures and other structural surfaces is remote sensing measurement technology, a three-dimensional real scene model of rock slope is constructed, and the structural surfaces distributed on the model are identified and interpreted. The remote sensing images commonly used for slope texture feature identification mainly include optical images, laser radar images and synthetic aperture radar images. Due to the complementarity of these images, the identification of texture features by single modal images has limitations. Therefore, it is necessary to integrate the texture information provided by different imaging methods, which has strong application demand in practice.

[0003] On the other hand, the segmentation accuracy of the existing remote sensing mapping technology for weak texture slope features is often restricted by factors such as light intensity and image noise. Therefore, it is necessary to study a multi-modal weak texture slope remote sensing image feature segmentation method. SUMMARY

[0004] In order to solve the above problems in the background art, the application provides a segmentation method for multi-modal weak texture slope remote sensing images based on SAM. The method realizes the efficient application of SAM in the segmentation task of multi-modal weak texture slope images by introducing a lightweight HRSI module and a modal perception module, and introducing a multi-scale attention mechanism in the image encoder.

[0005] The purpose of the application is realized by the following technical scheme:

[0006] A segmentation method for multi-modal weak texture slope remote sensing images based on SAM, comprising the following steps:

[0007] Step 1, constructing a weak texture slope segmentation model based on a SAM network, for segmenting the surface features and background parts of the slope;

[0008] Step 2, performing bilateral filtering and normalization processing on the multi-modal weak texture slope remote sensing image to be segmented;

[0009] Step 3, combining the Dice loss and cross-entropy loss to train the weak texture slope segmentation model based on the SAM network constructed in step 1, and freezing the parameters of the image encoder during the training process;

[0010] Step 4, input the multi-modal weak-texture slope remote sensing image processed in step 2 into the weak-texture slope segmentation model based on the SAM network trained in step 3, and the weak-texture slope segmentation model based on the SAM network outputs the segmentation result of the slope crack feature in the multi-modal weak-texture slope remote sensing image.

[0011] In the present application, the weak-texture slope segmentation model based on the SAM network comprises an image encoder, a prompt encoder, a mask decoder, an HRSI module and a modal perception module, wherein:

[0012] The HRSI module processes the input non-optical modal image to obtain the non-optical feature of the non-optical modal image;

[0013] The modal perception module interacts with the non-optical feature of the non-optical modal image obtained by the HRSI module and the optical image feature to complete the enhancement of the non-optical feature and input to the image encoder;

[0014] The image encoder is used to map the image to be segmented to an image feature space to obtain image features;

[0015] The prompt encoder comprises a first encoding module, a second encoding module and a third encoding module, and generates dense prompts, sparse prompts and image position encodings respectively according to the image features obtained by the image encoder;

[0016] The mask decoder predicts the segmentation mask by integrating the outputs of the image encoder and the prompt encoder;

[0017] In the present application, the HRSI module is composed of two convolution modules, a central difference convolution module, a maximum pooling layer, an up-sampling layer and an adaptive deep and shallow feature fusion module. The non-optical modal image I is first input into the first convolution module to initially extract image features to obtain feature F1; then the feature F1 is input into the central difference convolution module and the maximum pooling layer respectively. The central difference convolution module processes the feature F1 to obtain the shallow fine-grained feature F S , and the maximum pooling layer processes the feature F1 to generate the feature F m ; the feature F m is input into the second convolution module connected with the maximum pooling layer for feature extraction, and the second convolution module is output after being processed by the up-sampling layer to obtain the feature F d ; the feature F d and the feature F S are input into the adaptive deep and shallow feature fusion module to generate the fusion feature F fusion .

[0018] In this invention, a multi-scale feature extraction module is introduced into the image encoder. In the MobileNetV2 image encoder network, the outputs of the 5th Bottle-Neck and the 7th Bottle-Neck are selected to extract features at different scales. Then, the features of BottleNeck6 and BottleNeck7 are added together and upsampled by 2 times. Finally, they are fused with the features of BottleNeck5 as the output of BottleNeck5. The features of BottleNeck5 and BottleNeck7 after feature fusion are passed through the MRLA-Light module to strengthen the connection between different layers and obtain more discriminative features. These features are then input together with the features of BottleNeck5 into the pooling layer and the fully connected layer.

[0019] In this invention, the modality perception module employs two cross-modality attention modules to achieve the interaction between non-optical features of non-optical modal images and optical image features. The specific process is as follows:

[0020] The encoding of non-optical modal slope images is performed by the HRSI module g. Let the input non-optical modal slope image data be x. D Let its width and height be represented by H and W respectively. Then, the characteristic f of the non-optical modality slope image... D The acquisition method is as follows:

[0021] f D =g(x D );

[0022] For the non-optical features of the j-th layer and optical characteristics The formula for the cross-modal attention process is:

[0023]

[0024] Here, CMA(·,·) represents cross-modal attention computation. This represents the non-optical feature information that the optical mode focuses on, and the enhanced non-optical features. Represented as After passing through the j-th layer of the network, we obtain... This completes the enhancement from non-optical features to optical image features;

[0025] The enhancement process of non-optical features by optical image features is as follows:

[0026]

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] (1) The present application can process multi-modal weak texture slope image data such as optical-synthetic aperture radar images on the basis of the existing SAM model by integrating a lightweight modal perception module, greatly improving the recognition efficiency of the image.

[0029] (2) The present application improves the recognition and extraction ability of the model for weak texture by introducing a multi-scale attention module in the image encoder, ensuring the recognition accuracy of the model.

[0030] (3) Compared with the traditional neural network model, the present application occupies less computing resources and has faster model training speed, ensuring the efficiency of multi-modal weak texture slope image data processing. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 It is a flowchart of the multi-modal weak texture slope image segmentation method based on the SAM model.

[0032] Figure 2 It is a schematic diagram of the training network structure of the SAM model.

[0033] Figure 3 It is a schematic diagram of the HRSI module structure.

[0034] Figure 4 It is a schematic diagram of the image encoder structure.

[0035] Figure 5 It is a schematic diagram of the modal perception module structure.

[0036] Figure 6 It is a schematic diagram of the image segmentation process. DETAILED DESCRIPTION

[0037] The technical solutions of the present application will be further described below in conjunction with the drawings, but are not limited thereto, and any modification or equivalent replacement of the technical solutions of the present application without departing from the spirit and scope of the present application shall be covered in the protection scope of the present application.

[0038] The present application provides a multi-modal weak texture slope remote sensing image segmentation method based on SAM, as shown in Figure 1 and Figure 6 The method comprises the following steps:

[0039] Step 1, construct a weak texture slope segmentation model based on the SAM network, for segmenting the slope surface features and background parts, including an image encoder, a prompt encoder, a mask decoder, an HRSI module and a modal perception module, wherein:

[0040] The prompt encoder comprises a first encoding module, a second encoding module and a third encoding module, and is used for generating dense prompts, sparse prompts and image position encodings respectively;

[0041] The HRSI module (i.e., a modal encoder) encodes an input non-optical modal image to obtain non-optical features of the non-optical modal image, and the non-optical modal image is a SAR image, LiDAR point cloud data, a thermal infrared image, multi-modal fusion data or a non-standard format image.

[0042] The HRSI module processes the input SAR image to obtain SAR features of a weak-texture slope image of the SAR modal.

[0043] The modal perception module interacts with the optical image features according to the SAR features obtained by the HRSI module, completes enhancement of the SAR features and inputs the enhanced SAR features to the image encoder.

[0044] The image encoder is used for mapping an image to be segmented to an image feature space.

[0045] The mask decoder predicts a segmentation mask by integrating outputs of the image encoder and the prompt encoder, that is, the model predicts a region in the image that should be segmented according to a prompt word.

[0046] The present application adds a lightweight HRSI module and a modal perception module to the structure of the image encoder, the mask decoder and the prompt encoder of the existing SAM model, gives the SAM model multi-modal feature perception capability, and introduces a multi-scale feature extraction module in the image encoder, further improves the recognition performance of weak texture features.

[0047] As shown in Figure 3 The HRSI module mainly comprises a convolution module (CNN Block), a center difference convolution module (CDC Block), a maximum pooling layer, an up-sampling layer and an adaptive deep and shallow feature fusion (ADSFF) module. Specifically, the image I is first input into the first convolution module to initially extract image features. The convolution module comprises three layers, the first layer adopts Dropout with a parameter value of 0.2 to prevent the network from relying too much on local features and effectively avoid overfitting; the second layer is a two-dimensional convolution operation, which comprises 50 convolution kernels with a size of 3*3 to filter different features of the input image; the third layer is an activation function layer, which adopts a nonlinear activation function LeakyReLU with a parameter value of 0.3 for activation operation; after the first convolution module processing, the feature F1 is obtained. Then the feature F1 is input into the center difference convolution module and the maximum pooling layer respectively. The center difference convolution module processes the feature F1 to obtain a shallow fine-grained feature F SMax pooling layers process feature F1 to generate feature F. m This reduces the number of network parameters while preserving the discriminative features of the image; then feature F m The input is fed into a second convolutional module connected to it for feature extraction. The output of the second convolutional module is processed by an upsampling layer to obtain the feature F. d ; feature F d and F S The input is fed into the Adaptive Deep and Shallow Feature Fusion Module (ADSFF) to generate the fused feature F. fusion .

[0048] like Figure 4 As shown, to enhance feature information while maintaining the network's recognition speed, the multi-scale feature extraction module selects the outputs of the 5th and 7th Bottle-Necks in the MobileNetV2 image encoder network to extract features at different scales. Then, the features of BottleNeck6 and BottleNeck7 are added together and upsampled by a factor of 2. Finally, these features are fused with the features of BottleNeck5 to obtain the output of BottleNeck5. The resulting fused BottleNeck5 and BottleNeck7 features are then processed by the MRLA-Light module to strengthen the connections between different layers, obtaining more discriminative features. These features, along with the BottleNeck5 features, are then input into the pooling layer and the fully connected layer.

[0049] like Figure 5 As shown, the modal perception module uses two cross-modal attention modules to realize the interaction between SAR features and optical image features.

[0050] The slope image encoding for the SAR mode is performed by the HRSI module g. Let the input slope image data for the SAR mode be x. D Its width and height are represented by H and W respectively. Then, the characteristic f of the SAR mode slope image... D The acquisition method is as follows:

[0051] f D =g(x D ).

[0052] This invention utilizes a modal perception module to achieve interaction between SAR features and optical image features. This process is implemented through two cross-modal attention modules. To illustrate the modal interaction process, we will use SAR features from the j-th layer of the model as an example. and optical characteristics Taking the fusion process as an example, the formula for the cross-modal attention process is:

[0053]

[0054] Here CMA(·,·) represents the cross-modal attention calculation, represents the SAR feature (i.e., non-optical feature) information focused on by the optical modality, and the enhanced SAR feature may be represented as After passing through the network of the jth layer, we obtain

[0055] So far, the SAR-to-optical enhancement has been completed, and the following describes the enhancement of the SAR feature information by the optical modality, which is as follows:

[0056]

[0057] Step 2: Bilateral filtering and normalization processing are performed on the multi-modal weak-texture slope remote sensing image used for model training and subsequent identification in step 1, to remove noise and enhance texture features, reduce the influence of light, improve the quality of the multi-modal weak-texture remote sensing image, and improve the model training effect and image identification accuracy.

[0058] Step 3: The Dice loss and cross-entropy loss are combined to train the weak-texture slope segmentation model based on the SAM network constructed in step 1, and the parameters of the image encoder are frozen during the training process.

[0059] In the present application, the training process of the weak-texture slope segmentation model based on the SAM network is as shown in Figure 2 The cross-entropy loss and Dice loss are used to train the weak-texture slope segmentation model based on the SAM network, and the trained weak-texture slope segmentation model based on the SAM network is obtained. During the training process, the parameters of the image encoder are frozen to improve the training efficiency and reduce the demand for computing resources, greatly reducing the number of parameters in the model training.

[0060] In the present application, the cross-entropy loss L CE and the Dice loss L Dice are used to train the segmentation network:

[0061]

[0062] where H and W represent the height and width of the image, respectively, y represents the segmentation label, and y' represents the segmentation result of the model. The total training loss is the linear addition of L CE and L Dice .

[0063] Step 4: The multi-modal weak-texture slope remote sensing image processed in step 2 is input into the weak-texture slope segmentation model based on the SAM network trained in step 3, and the weak-texture slope segmentation model based on the SAM network outputs the segmentation result of the slope crack feature in the image.

[0064] The LiDAR point cloud data, the thermal infrared image, the multi-modal fusion data and the non-standard format image segmentation method is the same as the SAR image, and details are not repeated here.

[0065] In the present application, the weak texture slope segmentation model of the trained SAM network is used to output the segmentation results of each object in the multi-modal weak texture slope image according to the click prompt or the object frame prompt. For example, the multi-modal image to be segmented is an image obtained by different imaging methods, such as optical image-SAR, optical image-LiDAR point cloud composed of multi-modal slope image data. The segmentation result is input to the related slope image recognition model for subsequent recognition.

[0066] The present application avoids the huge computational overhead of retraining a large model for each multi-modal task, and realizes the feature encoding of optical modal information or other modal information by introducing a lightweight HRSI module and a modal perception module. The modal perception module injects the encoded optical modal information into the backbone network of the SAM model, realizes the alternative enhancement of SAR features and optical features, and thus realizes the multi-modal expansion of the SAM model.

Claims

1. A segmentation method for multimodal weakly textured slope remote sensing images based on SAM, characterized in that... The method includes the following steps: Step 1: Construct a weakly textured slope segmentation model based on a SAM network to segment the surface features and background of the slope. The SAM-based weakly textured slope segmentation model includes an image encoder, a cue encoder, a mask decoder, an HRSI module, and a modality sensing module, wherein: The HRSI module processes the input non-optical modal image to obtain its non-optical features; The modal perception module interacts with the optical image features of the non-optical modal image obtained by the HRSI module to enhance the non-optical features and input them to the image encoder. The image encoder is used to map the image to be segmented to an image feature space to obtain image features; The prompt encoder includes a first encoding module, a second encoding module, and a third encoding module, which generate dense prompts, sparse prompts, and image position codes respectively based on the image features obtained by the image encoder; The mask decoder predicts a segmentation mask by integrating the outputs of the image encoder and the cue encoder. The HRSI module consists of two convolutional modules, a central difference convolutional module, a max pooling layer, an upsampling layer, and an adaptive deep and shallow feature fusion module for non-optical modal images. The data is first fed into the first convolutional module to initially extract image features, thus obtaining the features. Then the features The inputs are fed into the central difference convolutional module and the max pooling layer, respectively. The central difference convolutional module processes the features. Then, fine-grained features of the shallow layer were obtained. Maximum pooling layer processing features To generate features ;feature The input is fed into a second convolutional module connected to the max-pooling layer for feature extraction. The output of the second convolutional module is processed by an upsampling layer to obtain the features. ; Features and characteristics The input is fed into the adaptive deep and shallow feature fusion module to generate fused features. ; Step 2: Perform bilateral filtering and normalization on the remote sensing image of the multimodal weakly textured slope to be segmented; Step 3: Combine Dice loss and cross-entropy loss to train the weakly textured slope segmentation model based on the SAM network built in Step 1, and freeze the parameters of the image encoder during training. Step 4: Input the multimodal weak textured slope remote sensing image processed in Step 2 into the weak textured slope segmentation model based on the SAM network trained in Step 3. The weak textured slope segmentation model based on the SAM network outputs the segmentation results of slope crack features in the multimodal weak textured slope remote sensing image.

2. The segmentation method for multimodal weakly textured slope remote sensing images based on SAM according to claim 1, characterized in that... The image encoder introduces a multi-scale feature extraction module. In the MobileNetV2 image encoder network, the outputs of the 5th Bottle-Neck and the 7th Bottle-Neck are selected to extract features at different scales. Then, the features of BottleNeck6 and BottleNeck7 are added together and upsampled by 2 times. Finally, they are fused with the features of BottleNeck5 as the output of BottleNeck5. The fused features of BottleNeck5 and BottleNeck7 are then processed by the MRLA-Light module to strengthen the connection between different layers and obtain more discriminative features. These features are then input together with the features of BottleNeck5 into the pooling layer and the fully connected layer.

3. The segmentation method for multimodal weakly textured slope remote sensing images based on SAM according to claim 1, characterized in that... The modality perception module employs two cross-modal attention modules to achieve the interaction between non-optical features of non-optical modal images and optical image features. The specific process is as follows: Non-optical modal slope image encoding is performed by the HRSI module. Complete. Let the input non-optical modal slope image data be... Its width and height are respectively... , This indicates that the characteristics of non-optical modal slope images... The acquisition method is as follows: ； For the Non-optical features of the layer and optical characteristics The formula for the cross-modal attention process is: Here Represents cross-modal attention computation. This represents the non-optical feature information that the optical mode focuses on, and the enhanced non-optical features. Represented as , Through the first After obtaining the network of layers This completes the enhancement from non-optical features to optical image features; The enhancement process of non-optical features by optical image features is as follows: 。 4. The segmentation method for multimodal weakly textured slope remote sensing images based on SAM according to claim 1, characterized in that... The non-optical modal images include SAR images, LiDAR point cloud data, thermal infrared images, multimodal fusion data, and non-standard format images.

5. The segmentation method for multimodal weakly textured slope remote sensing images based on SAM according to claim 1, characterized in that... The cross-entropy loss and Dice loss for: ； ； in, and These represent the height and width of the image, respectively. Indicates a separator label. This represents the segmentation result of the model.

Images