Multi-modal weak texture slope remote sensing image segmentation method based on SAM

Abstract

The invention discloses a multi-mode weak texture slope remote sensing image segmentation method based on SAM. The method comprises the following steps: step 1, constructing a weak texture slope segmentation model based on an SAM network; step 2, performing bilateral filtering and normalization processing on the multi-mode weak texture slope remote sensing image; 3, training a weak texture slope segmentation model by using Dice loss and cross entropy loss in a combined manner, and freezing parameters of an image encoder in the training process; and step 4, inputting the processed multi-modal weak-texture slope remote sensing image into the trained weak-texture slope segmentation model, and outputting a segmentation result of slope crack features in the multi-modal weak-texture slope remote sensing image based on the weak-texture slope segmentation model of the SAM network. According to the method, a lightweight HRSI module and a modal perception module are introduced, and a multi-scale attention mechanism is introduced into an image encoder, so that efficient application of SAM in a multi-modal weak texture slope image segmentation task is realized.

Description

Technical Field

[0001] The invention belongs to the technical field of surveying and mapping, and relates to a texture feature segmentation method, and in particular to a texture feature segmentation method applied to multi-modal weak-texture slope remote sensing images. Background Art

[0002] At present, the main means of quickly obtaining key information on rock mass joints, cracks and other structural surfaces is to use remote sensing measurement technology to construct a three-dimensional real-life model of the rock slope, and then identify and interpret the structural surfaces distributed on the model. Remote sensing images commonly used for slope texture feature identification mainly include optical images, lidar images, synthetic aperture radar images, etc. Due to the complementarity of these images, single-mode images have limitations in identifying texture features. Therefore, there is a strong application demand in practice to integrate the texture information provided by different imaging methods.

[0003] On the other hand, the segmentation accuracy of existing remote sensing mapping technology for weak texture slope features is often restricted by factors such as light intensity and image noise. Therefore, it is necessary to study a method for multimodal weak texture slope remote sensing image feature segmentation. Summary of the invention

[0004] In order to solve the above problems existing in the background technology, the present invention provides a multimodal weak texture slope remote sensing image segmentation method based on SAM. The method realizes the efficient application of SAM in the multimodal weak texture slope image segmentation task by introducing a lightweight HRSI module and a modal perception module, and introducing a multi-scale attention mechanism in the image encoder.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A method for segmenting a multimodal weak-texture slope remote sensing image based on SAM comprises the following steps:

[0007] Step 1: construct a weak texture slope segmentation model based on SAM network to segment the slope surface features and background parts;

[0008] Step 2, performing bilateral filtering and normalization processing on the multimodal weak texture slope remote sensing image to be segmented;

[0009] Step 3: Combine the Dice loss and the cross entropy loss to train the weak texture slope segmentation model based on the SAM network constructed in step 1, and freeze the parameters of the image encoder during the training process;

[0010] Step 4: Input the multimodal weak texture slope remote sensing image processed in step 2 into the weak texture slope segmentation model based on the SAM network trained in step 3. The weak texture slope segmentation model based on the SAM network outputs the segmentation result of the slope crack features in the multimodal weak texture slope remote sensing image.

[0011] In the present invention, the weak texture slope segmentation model based on the SAM network includes an image encoder, a prompt encoder, a mask decoder, a HRSI module and a modality perception module, wherein:

[0012] The HRSI module processes the input non-optical modality image to obtain non-optical features of the non-optical modality image;

[0013] The modality perception module interacts with the optical image features according to the non-optical features of the non-optical modality image acquired by the HRSI module, completes the enhancement of the non-optical features and inputs them into the image encoder;

[0014] The image encoder is used to map the image to be segmented into an image feature space to obtain image features;

[0015] The hint encoder comprises a first encoding module, a second encoding module and a third encoding module, which respectively generate dense hints, sparse hints and image position codes according to the image features obtained by the image encoder;

[0016] The mask decoder predicts the segmentation mask by integrating the outputs of the image encoder and the hint encoder;

[0017] In the present invention, the HRSI module is composed of two convolution modules, a center difference convolution module, a maximum pooling layer, an upsampling layer and an adaptive deep and shallow feature fusion module. The non-optical modality image I is first input into the first convolution module to initially extract image features to obtain feature F1; then the feature F1 is input into the center difference convolution module and the maximum pooling layer respectively, and the center difference convolution module processes the feature F1 to obtain the shallow fine-grained feature F S , the maximum pooling layer processes feature F1 to produce feature F m ; Feature F m The input is sent to the second convolution module connected to the maximum pooling layer for feature extraction. The output of the second convolution module is processed by the upsampling layer to obtain the feature F d ; The feature F d and feature F S Input to the adaptive deep and shallow feature fusion module to generate fusion feature F fusion .

[0018] In the present invention, a multi-scale feature extraction module is introduced into the image encoder, and the outputs of the 5th Bottle-Neck and the 7th Bottle-Neck are selected in the image encoder MobileNetV2 network to extract features of different scales, and then the features of BottleNeck6 and BottleNeck7 are added and up-sampled by a factor of 2, and finally fused with the features of BottleNeck5 as the output of BottleNeck5. The fused BottleNeck5 features and BottleNeck7 features are passed through the MRLA-Light module to strengthen the connection between different layers and obtain more discriminative features, and are input into the pooling layer and the fully connected layer together with the features of BottleNeck5.

[0019] In the present invention, the modality perception module uses two cross-modality attention modules to realize the interaction between the non-optical features of the non-optical modality image and the optical image features. The specific process is as follows:

[0020] The non-optical slope image encoding is completed by the HRSI module g. Suppose the input non-optical slope image data is x D , its width and height are represented by H and W respectively, then the feature f of the slope image of the non-optical mode is D The acquisition method is as follows:

[0021] f D =g(x D );

[0022] For the non-optical features of layer j and optical characteristics The cross-modal attention process formula is:

[0023]

[0024] Here CMA(·,·) represents cross-modal attention computation, Indicates the non-optical feature information that the optical modality pays attention to, and the enhanced non-optical features Expressed as After passing through the j-th layer of the network, we get At this point, the enhancement from non-optical features to optical image features is completed;

[0025] The process of enhancing non-optical features by optical image features is as follows:

[0026]

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] (1) By integrating a lightweight modal perception module, the present invention can process multi-modal weak-texture slope image data such as optical-synthetic aperture radar images on the basis of the existing SAM model, thereby greatly improving the image recognition efficiency.

[0029] (2) The present invention introduces a multi-scale attention module into the image encoder, thereby improving the model's ability to recognize and extract weak textures, thereby ensuring the recognition accuracy of the model.

[0030] (3) Compared with the traditional neural network model, the present invention occupies less computing resources and has a faster model training speed, ensuring the high efficiency of multimodal weak texture slope image data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of the multimodal weak texture slope image segmentation method based on the SAM model.

[0032] Figure 2 Schematic diagram of the training network structure of the SAM model.

[0033] Figure 3 Figure 2 is a schematic diagram of the HRSI module structure.

[0034] Figure 4 This is a schematic diagram of the image encoder structure.

[0035] Figure 5 This is a schematic diagram of the modal perception module structure.

[0036] Figure 6 Schematic diagram of the image segmentation process. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0038] The present invention provides a multi-modal weak texture slope remote sensing image segmentation method based on SAM, such as Figure 1 and Figure 6 As shown, the method comprises the following steps:

[0039] Step 1: Construct a weak texture slope segmentation model based on the SAM network to segment the surface features and background of the slope, including an image encoder, a hint encoder, a mask decoder, a HRSI module and a modality perception module, where:

[0040] The hint encoder comprises a first encoding module, a second encoding module and a third encoding module, which are used to generate dense hints, sparse hints and image position encoding respectively;

[0041] The HRSI module (i.e., modality encoder) encodes the input non-optical modality image to obtain the non-optical features of the non-optical modality image. The non-optical modality image is a SAR image, LiDAR point cloud data, thermal infrared image, multi-modal fusion data, and non-standard format image. The SAR image is used as an example for explanation below.

[0042] The HRSI module processes the input SAR image to obtain SAR features of the weak texture slope image of the SAR mode;

[0043] The modality perception module interacts with the optical image features according to the SAR features acquired by the HRSI module, enhances the SAR features and inputs them into the image encoder;

[0044] The image encoder is used to map the image to be segmented into an image feature space;

[0045] The mask decoder predicts the segmentation mask, i.e., the area in the image that the model should segment based on the cue word, by integrating the outputs of the image encoder and the cue encoder.

[0046] The present invention adds a lightweight HRSI module and a modal perception module to the structure of the existing SAM model's image encoder, mask decoder, and prompt encoder, endows the SAM model with multi-modal feature perception capabilities, and introduces a multi-scale feature extraction module into the image encoder, further improving the recognition performance of weak texture features.

[0047] like Figure 3 As shown in the figure, the HRSI module is mainly composed of a convolution module (CNN Blocks), a center difference convolution module (CDCBlock), a maximum pooling layer, an upsampling layer, and an adaptive deep and shallow feature fusion (ADSFF) module. Specifically, image I is first input into the first convolution module to initially extract image features. Among them, the convolution module contains three layers. The first layer uses Dropout with a parameter value of 0.2 to prevent the network from being overly dependent on local features and effectively avoid overfitting; the second layer is a two-dimensional convolution operation, which contains 50 convolution kernels of size 3×3 to filter different features of the input image; the third layer is an activation function layer, which uses a nonlinear activation function LeakyReLU with a parameter value of 0.3 for activation operation; after being processed by the first convolution module, feature F1 is obtained. Feature F1 is then input into the center difference convolution module and the maximum pooling layer respectively. After the center difference convolution module processes feature F1, a shallow fine-grained feature F is obtained. S; The maximum pooling layer processes feature F1 to produce feature F m , reducing the number of network parameters and retaining the distinguishing features of the image; then the feature F m The input is sent to the second convolution module connected to it for feature extraction. The output of the second convolution module is processed by the upsampling layer to obtain the feature F d ; The feature F d and F S Input to the adaptive deep and shallow feature fusion module (ADSFF) to generate the fusion feature F fusion .

[0048] like Figure 4 As shown, in order to enhance feature information while ensuring the recognition speed of the network, the multi-scale feature extraction module selects the output of the 5th and 7th Bottle-Neck in the image encoder MobileNetV2 network to extract features of different scales, then adds the features of BottleNeck6 and BottleNeck7 and performs 2x upsampling, and finally fuses them with the features of BottleNeck5 as the output of BottleNeck5. The obtained BottleNeck5 features and BottleNeck7 features after feature fusion are passed through the MRLA-Light module to strengthen the connection between different layers, obtain more discriminative features, and are input into the pooling layer and the fully connected layer together with the features of BottleNeck5.

[0049] like Figure 5 As shown, the modal perception module uses two cross-modal attention modules to realize the interaction between SAR features and optical image features.

[0050] The slope image encoding of SAR mode is completed by HRSI module g. Assume that the input slope image data of SAR mode is x D , its width and height are represented by H and W respectively, then the characteristic f of the slope image of SAR mode is D The acquisition method is as follows:

[0051] f D =g(x D ).

[0052] The present invention uses the modality perception module to realize the interaction between SAR features and optical image features. This process is realized through two cross-modality attention modules. To illustrate the modality interaction process, the SAR features of the jth layer of the model are used here. and optical characteristics Taking the fusion process of as an example, the formula of the cross-modal attention process is:

[0053]

[0054] Here CMA(·,·) represents cross-modal attention computation, Indicates the SAR feature information (i.e. non-optical feature) that the optical mode pays attention to, and the enhanced SAR feature It can be expressed as After passing through the j-th layer of the network, we get

[0055] So far, the enhancement from SAR to optics has been completed. Next, we will introduce the enhancement of SAR feature information from optics. The process is as follows:

[0056]

[0057] Step 2: Perform bilateral filtering and normalization on the multimodal weak-texture slope remote sensing images used in the model training and subsequent recognition in step 1 to remove noise and enhance texture features, while reducing the impact of illumination, improving the quality of multimodal weak-texture remote sensing images, and improving the model training effect and image recognition accuracy.

[0058] Step 3: Combine the Dice loss and cross entropy loss to train the SAM network-based weak texture slope segmentation model constructed in step 1, and freeze the parameters of the image encoder during the training process.

[0059] In the present invention, the training process of the weak texture slope segmentation model based on the SAM network is as follows: Figure 2 As shown in the figure, the cross entropy loss and Dice loss are used to train the weak texture slope segmentation model based on the SAM network to obtain the trained weak texture slope segmentation model based on the SAM network. During the training process, the parameters of the image encoder are frozen to improve the training efficiency and reduce the computing resource requirements, greatly reducing the number of parameters for model training.

[0060] In the present invention, the cross entropy loss L is used CE and Dice loss L Dice To train the segmentation network:

[0061]

[0062] Where H and W represent the height and width of the image, y represents the segmentation label, and y' represents the segmentation result of the model. The total training loss is L CE and L Dice The linear addition of .

[0063] Step 4: Input the multimodal weak texture slope remote sensing image processed in step 2 into the weak texture slope segmentation model based on the SAM network trained in step 3; the weak texture slope segmentation model based on the SAM network outputs the segmentation result of the slope crack features in the image.

[0064] The segmentation methods for LiDAR point cloud data, thermal infrared images, multimodal fusion data, and non-standard format images are the same as those for SAR images and will not be repeated here.

[0065] In the present invention, the trained weak texture slope segmentation model of the SAM network is used to output the segmentation results of each object in the multimodal weak texture slope image according to the click prompt or the object frame prompt. For example, the images of multiple modes to be segmented are images obtained by different imaging methods, such as optical image-SAR, optical image-LiDAR point cloud composed of multimodal slope image data. The segmentation results are input to the recognition model of the relevant slope image for subsequent recognition.

[0066] The present invention avoids the huge computational overhead of retraining a large model for each multimodal task. By introducing a lightweight HRSI module and a modal perception module, feature encoding of optical modal information or other modal information is achieved. The modal perception module injects the encoded optical modal information into the backbone network of the SAM model to achieve alternating enhancement of SAR features and optical features, thereby realizing multimodal expansion of the SAM model.

Claims

1. A multi-modal weak texture slope remote sensing image segmentation method based on SAM, characterized by The method comprises the following steps: Step 1: construct a weak texture slope segmentation model based on the SAM network, which is used to segment the slope surface features and the background part. The weak texture slope segmentation model based on the SAM network includes an image encoder, a prompt encoder, a mask decoder, a HRSI module and a modal perception module, wherein: The HRSI module processes the input non-optical modality image to obtain non-optical features of the non-optical modality image; The modality perception module interacts with the optical image features according to the non-optical features of the non-optical modality image acquired by the HRSI module, completes the enhancement of the non-optical features and inputs them into the image encoder; The image encoder is used to map the image to be segmented into an image feature space to obtain image features; The hint encoder comprises a first encoding module, a second encoding module and a third encoding module, which respectively generate dense hints, sparse hints and image position codes according to the image features obtained by the image encoder; The mask decoder predicts the segmentation mask by integrating the outputs of the image encoder and the hint encoder; Step 2, performing bilateral filtering and normalization processing on the multimodal weak texture slope remote sensing image to be segmented; Step 3: Combine the Dice loss and the cross entropy loss to train the weak texture slope segmentation model based on the SAM network constructed in step 1, and freeze the parameters of the image encoder during the training process; Step 4: Input the multimodal weak texture slope remote sensing image processed in step 2 into the weak texture slope segmentation model based on the SAM network trained in step 3. The weak texture slope segmentation model based on the SAM network outputs the segmentation result of the slope crack features in the multimodal weak texture slope remote sensing image.

2. The method for segmenting multimodal weak texture slope remote sensing images based on SAM according to claim 1 is characterized in that The HRSI module consists of two convolution modules, a center difference convolution module, a maximum pooling layer, an upsampling layer and an adaptive deep and shallow feature fusion module. The non-optical modality image I is first input into the first convolution module to initially extract image features and obtain feature F1; then feature F1 is input into the center difference convolution module and the maximum pooling layer respectively. After the center difference convolution module processes feature F1, the shallow fine-grained feature F S , the maximum pooling layer processes feature F1 to produce feature F m ; Feature F m The input is sent to the second convolution module connected to the maximum pooling layer for feature extraction. The output of the second convolution module is processed by the upsampling layer to obtain the feature F d ; The feature F d and feature F S Input to the adaptive deep and shallow feature fusion module to generate fusion feature F fusion .

3. The method for segmenting multimodal weak texture slope remote sensing images based on SAM according to claim 1 is characterized in that A multi-scale feature extraction module is introduced into the image encoder. The outputs of the 5th Bottle-Neck and the 7th Bottle-Neck are selected in the image encoder MobileNetV2 network to extract features of different scales. Then, the features of BottleNeck6 and BottleNeck7 are added and up-sampled by a factor of 2. Finally, the features are fused with the features of BottleNeck5 as the output of BottleNeck5. The fused features of BottleNeck5 and BottleNeck7 are passed through the MRLA-Light module to strengthen the connection between different layers and obtain more discriminative features. The features are input into the pooling layer and the fully connected layer together with the features of BottleNeck5.

4. The method for segmenting multimodal weak texture slope remote sensing images based on SAM according to claim 1, characterized in that The modality perception module uses two cross-modal attention modules to realize the interaction between the non-optical features of non-optical modality images and the optical image features. The specific process is as follows: The non-optical slope image encoding is completed by the HRSI module g. Suppose the input non-optical slope image data is x D , its width and height are represented by H and W respectively, then the feature f of the slope image of the non-optical mode is D The acquisition method is as follows: f D =g(x D ); For the non-optical features of layer j and optical characteristics The cross-modal attention process formula is: Here CMA(·,·) represents cross-modal attention computation, Indicates the non-optical feature information that the optical modality pays attention to, and the enhanced non-optical features Expressed as After passing through the j-th layer of the network, we get At this point, the enhancement from non-optical features to optical image features is completed; The process of enhancing non-optical features by optical image features is as follows:

5. The method for segmenting multimodal weak texture slope remote sensing images based on SAM according to claim 1, characterized in that The non-optical modality images are SAR images, LiDAR point cloud data, thermal infrared images, multi-modal fusion data and non-standard format images.

6. The method for segmenting multimodal weak texture slope remote sensing images based on SAM according to claim 1, characterized in that The cross entropy loss L CE and Dice loss L Dice for: Among them, H and W represent the height and width of the image respectively, y represents the segmentation label, and y' represents the segmentation result of the model.

Images