Remote sensing image interactive segmentation method based on visual base large model fine tuning

Abstract

The application discloses a kind of based on visual basic large model fine-tuning remote sensing image interactive segmentation method, its core model includes image encoder with adaptive layer, prompt encoder, mask decoder, realizes complete human-computer interaction cycle, including image band conversion, image encoder original parameter freezing and adaptive layer parameter training, prompt encoder receives simulation click input and parameter training, mask decoder parameter training and feedback, prompt simulator simulates click generation, prompt encoder receives user click input, mask decoder predicts mask output.The application has established complete practical interactive remote sensing image segmentation training and reasoning process, iteratively outputs or corrects predicted mask by user click interaction, improves the accuracy and efficiency of remote sensing image segmentation, provides new method for practical efficient remote sensing image labeling.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image segmentation, and particularly relates to a remote sensing image interactive segmentation method based on visual basic large model fine tuning. BACKGROUND

[0002] With the continuous development of remote sensing technology, remote sensing satellites are continuously launched, and it is possible to collect massive remote sensing satellite image data, such as the high-resolution satellite which can capture a large number of high-resolution remote sensing images in the global range. In this era of remote sensing big data, a large amount of remote sensing image data brings a large number of application opportunities, and earth observation projects gradually increase, and at the same time, many challenges are also put forward; among these challenges, image segmentation of satellite images has become one of the most concerned basic problems, because it is a key technology widely used in city monitoring, city management, traffic management, agriculture, automatic mapping and navigation and other fields, and has a very important role.

[0003] Traditional image segmentation algorithms have an upper limit on performance, and expert knowledge is not fully integrated, making it difficult to directly apply the results. Remote sensing image datasets still rely on manual annotation. Previous interactive image segmentation methods have two main development directions: improving inference prediction speed to quickly obtain predicted segmentation results, such as backpropagation refinement scheme (BRS) from Jang WD, Kim C S. Interactive image segmentation via backpropagating refinement scheme [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 5297-5306, to f-BRS only backpropagating refinement at specific layers [Sofiiuk K, Petrov I, Barinova O, et al. f-brs: Rethinking backpropagating refinement for interactive segmentation [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 8623-8632], to RITM's pure feedforward network [Sofiiuk K, Petrov I A, Konushin A. Reviving iterative training with mask guidance for interactive segmentation [C] / / 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2022: 3141-3145], which greatly improves inference speed; improve the segmentation performance and robustness of the method to obtain high-quality and high-accuracy predicted segmentation results, such as Segment Anything Model [Kirillov A, Mintun E, Ravi N, et al. Segment anything [J]. arXiv preprint arXiv:2304.02643, 2023] uses a super large dataset to train the model, which has impressive segmentation ability and zero-shot transfer ability. In the implementation process, researchers found that visual foundation large models such as Segment Anything Model cannot accurately and efficiently implement remote sensing image segmentation, and the potential in the remote sensing field still needs to be tapped.

[0004] Therefore, how to fully utilize the segmentation capability of the visual base large model, adapt the visual base large model to the field of remote sensing, and further improve the segmentation effect and efficiency by combining expert knowledge through human-computer interaction has become an important problem. SUMMARY

[0005] In view of the above, the present application provides a remote sensing image interactive segmentation method based on visual base large model fine-tuning, which can fully utilize the performance of the visual base large model through joint fine-tuning strategy, and gradually correct the user's desired target object mask through human-computer interaction, thereby improving the accuracy and efficiency of remote sensing image segmentation.

[0006] A remote sensing image interactive segmentation method based on visual base large model fine-tuning, comprising the following steps:

[0007] (1) Obtain the visual base large model and its pre-trained model parameter file, and obtain the target remote sensing data set and preprocess the remote sensing images therein; the visual base large model comprises an image encoder, a prompt encoder and a mask decoder;

[0008] (2) Fine-tune the image encoder structure, that is, increase the adaptive layer to improve its feature extraction capability; during training, freeze the pre-training parameters of the image encoder, train the adaptive layer parameters using the target remote sensing data set, input the preprocessed remote sensing image into the image encoder, and obtain the image embedding vector;

[0009] (3) Generate positive clicks and negative clicks as interactive prompt input into the prompt encoder using the prompt simulator, and obtain the prompt embedding vector;

[0010] (4) Fine-tune the mask decoder structure, input the image embedding vector and the prompt embedding vector into the mask decoder, and obtain the predicted mask;

[0011] (5) Design an appropriate loss function to train the visual base large model;

[0012] (6) Input the remote sensing image to be segmented and user click interaction information into the trained model, and directly output the corresponding predicted mask as the image segmentation result.

[0013] Further, the visual base model obtained in step (1) is Segment Anything Model, and is pre-trained using a SA-1B dataset, wherein the image encoder is used to extract features of the image to obtain an embedding vector containing image feature information, the prompt encoder is used to interactively process the image to obtain an embedding vector containing prompt feature information, and the mask decoder is used to perform mask processing on the two embedding vectors to obtain a predicted mask, thereby realizing interactive segmentation of the image.

[0014] Further, the target remote sensing dataset includes to-be-labeled images, to-be-corrected mask images, collected remote sensing images, and interactive point coordinates (coordinates generated by the user clicking the remote sensing image).

[0015] Further, the pre-processing manner of the remote sensing image in step (1) is as follows: the RGB-IR four-band remote sensing image is converted into an RGB three-channel remote sensing image through band selection by using a front wave band converter, wherein a three-channel image is first extracted from the RGB band, and another three-channel image is obtained by replacing the R band in the original image with the IR band; during training, the two three-channel images are simultaneously input into the training to ensure that all input image data are processed in the same format.

[0016] Further, the image encoder uses a visual Transformer pre-trained by MAE (Masked Auto Encoder), which is composed of 12 ViT units connected in series, and each ViT unit is parallelly increased with an adaptive layer; each ViT unit is composed of LayerNorm, a multi-head attention mechanism layer, LayerNorm, and a multi-layer perception connected in sequence; the output of the adaptive layer is combined with the output of the multi-layer perception; the adaptive layer includes two low-rank parameter matrices M down and M up ; the input vector is first multiplied by the matrix M down for down-projection, then activated by ReLU, multiplied by the matrix M up for up-projection, and finally multiplied by a scaling coefficient f scale and output; during training, the pre-training parameters of the image encoder are frozen, and only the values of the matrices M down and M up are adjusted.

[0017] Further, the step (3) initializes the hint simulator at the first training, which simulates user interaction to obtain positive clicks or negative clicks, that is, simulates interactive sampling to sample a pixel or a solid circle with a suitable radius from the ground truth mask to generate a positive click, which is a click for segmenting the target object mask; simulates interactive sampling to sample a pixel or a solid circle with a suitable radius from the background to generate a negative click, which is a click for removing the mask not belonging to the target object; at the non-first training, the hint simulator simulates interactive sampling of a pixel or a solid circle with a suitable radius in the true negative area covered by the ground truth mask and not covered by the predicted mask as a positive click, and simulates interactive sampling of a pixel or a solid circle with a suitable radius in the false positive area not covered by the ground truth mask and covered by the predicted mask as a negative click; the hint simulator inputs all the clicks generated by it into the hint encoder to generate hint embedding vectors through position encoding.

[0018] Further, the simulated interactive sampling is designed according to the strategies of the aggregation, saliency, edge, and maximum distinguishability of artificial clicks.

[0019] Further, the mask decoder is composed of a hint self-attention mechanism layer, a hint-image cross-attention mechanism layer, an image vector upsampling layer, and a multi-layer perception connected in sequence to obtain the probability of each pixel becoming a mask, and then obtain the predicted mask through threshold control; the image vector upsampling layer is composed of a transpose convolution layer, a LayerNorm, a GELU activation function, a transpose convolution layer, and a GELU activation function connected in sequence.

[0020] Further, the expression of the loss function is as follows:

[0021]

[0022]

[0023]

[0024] Wherein: L is the total loss function, L CE is the cross-entropy loss, L FL is the focus loss, s i is the confidence of the i-th pixel point in the predicted mask, g i is the confidence of the i-th pixel point in the ground truth mask, γ is an adjustable factor, and N is the total number of pixel points in the mask.

[0025] The method of the present invention is a complete interactive segmentation process of remote sensing images based on fine-tuning of a large visual basic model. The core model includes an image encoder with an adaptation layer, a prompt encoder, and a mask decoder, realizing a complete human-computer interaction cycle, including image band conversion, freezing of original parameters of the image encoder and parameter training of the adaptation layer, prompt encoder receiving simulated click input and parameter training, mask decoder parameter training and feedback, prompt simulator simulating click generation, prompt encoder receiving user click input, and mask decoder predicting mask output.

[0026] The present invention proposes a joint fine-tuning strategy based on a large visual foundation model, which fine-tunes the adaptation layer of the image encoder, the prompt encoder, and the mask decoder simultaneously during training, making the model more adaptable to the target remote sensing dataset. The present invention proposes a weighted combination loss function of cross entropy and focus loss to calculate the loss of the predicted mask and the true value mask, and adjusts the parameters that need to be fine-tuned according to the loss. In order to support RGB-IR four-band and RGB three-band remote sensing images, the present invention uses a four-band to three-band pre-band converter to ensure that the image input to the image encoder is a three-channel image. In order to make the prompt encoder trainable, the present invention proposes an automated iterative feedback prompt simulation strategy, which generates the next positive and negative click pair based on the simulated interactive sampling and the difference between the predicted mask and the true value mask to simulate the segmentation click interaction.

[0027] The present invention establishes a complete and practical interactive remote sensing image segmentation training and reasoning process. Through user click interaction, iterative output or correction of prediction masks, improves the accuracy and efficiency of remote sensing image segmentation, and provides a new method for practical and efficient remote sensing image annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the steps of the interactive segmentation method of remote sensing images of the present invention.

[0029] Figure 2 Schematic diagram of the structure of the visual basic model of the present invention. DETAILED DESCRIPTION

[0030] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] like Figure 1 As shown, the interactive segmentation method of remote sensing images based on fine-tuning of a visual basic large model of the present invention includes the following steps:

[0032] (1) Obtain a visual foundation large model based on the "image encoder-prompt encoder-mask decoder" structure, obtain the model parameter file pre-trained by the SA-1B data set, so as to fine-tune. Obtain the target remote sensing data set, which contains images to be labeled, mask images to be corrected, interaction point coordinates, and remote sensing images collected by remote sensing equipment, and ensure the quality and diversity of the data.

[0033] The pre-training data set used in the embodiment is the SA-1B data set, the visual foundation large model uses the SegmentAnything Model, uses the AIRS remote sensing image data set with building roof labels as the 3-channel target remote sensing data set, and cuts the size to 512*512 pixels, retains the RGB 3 channels, and the number of images and corresponding building labels is 40000 respectively; use the ISPRS Potsdam remote sensing image data set with building labels as the 4-channel target remote sensing data set, and cut the size to 512*512 pixels, retain the RGB-IR 4 channels, and the number of images and corresponding building labels is 4000 respectively.

[0034] (2) Training phase, build a fine-tuning architecture for the foundation large model, as shown in Figure 2 , ensure that it includes image encoder adaptation layer and band converter, prompt encoder and simulator, and mask decoder part.

[0035] In the initial stage, the target remote sensing data set is used to train the whole model to adapt to specific remote sensing tasks, and the front band converter is used for pre-processing of the remote sensing image, that is, the band selection is performed on the RGB-IR four-band image to convert it into an RGB three-channel image, ensuring that all input data are processed in the same format. The band converter supports RGB three-band and RGB-IR four-band remote sensing images, the image encoder only supports RGB three-channel images due to the frozen pre-training parameters, and the order must be "RGB" or "BGR", the remote sensing image is converted into a three-channel remote sensing image through a band converter before inputting the image encoder; the conversion algorithm is to extract the RGB band to get a three-channel picture, and extract the IR band to replace the R band of the original image to get another three-channel picture; during training, the two pictures are sent into the training at the same time, during inference, the user selects three bands from the four bands to get a three-channel picture for inference, and the image after adjusting the number of bands is fed back to the user.

[0036] In this embodiment, the image encoder network structure can adopt MAE pre-trained ViT-B, the ViT batch size is 16, three-channel pictures need to be input, the learning rate is 1e-4, the weight decay is 0.01, the optimization algorithm is AdamW, and the expected fine-tuning model can be obtained after 100 epochs of training. During training, the RGB-IR picture is converted into an RGB three-channel picture and an IR-GB three-channel picture, and the two pictures are simultaneously input into training; during inference, the RGB three-channel is selected by default for inference, and the user can also select the three channels by himself.

[0037] (3) Training phase, fine-tuning the image encoder, gradually increasing the adaptive layer to improve its feature extraction capability, and the parameters of these adaptive layers are adjusted according to the low-rank parameter matrix to realize dimension transformation.

[0038] In this embodiment, the image encoder is a visual Transformer (ViT) pre-trained by MAE, which is composed of 12 ViT units, and each ViT unit increases an adaptive layer in parallel; the structure of each ViT unit is LayerNorm, multi-head attention, LayerNorm, and multi-layer perception; the input of the adaptive layer is the input of the first LayerNorm, and the output of the adaptive layer is combined with the output of the multi-layer perception; the adaptive layer includes two low-rank parameter matrices M down and M up , the input vector is multiplied by the first matrix M down to reduce the dimension (down-projection), and after ReLU activation, it is multiplied by the second matrix M up to increase the dimension (up-projection), and finally multiplied by a scaling factor f scale output; during training, the original image encoder parameters are frozen, and only the matrix values are adjusted; the structure of the multi-layer perception is a linear layer, a GELU activation function, and a linear layer; the image encoder vector dimension is 756, the image encoder depth is 12, the number of attention heads is 12, and the image vector length = image size / ViT batch size, which is 32.

[0039] (4) Training phase, generate positive clicks and negative clicks through the prompt simulator for the training of the encoder.

[0040] The prompt simulator in the embodiment simulates positive clicks or negative clicks obtained through user interaction, the positive clicks are clicks for segmenting a target object mask, and the negative clicks are clicks for removing a mask not belonging to the target object; the prompt simulator is initialized when the prompt simulator is trained for the first time, a positive click is generated from a pixel or a solid circle with a suitable radius randomly sampled from the ground truth mask, a negative click is generated from a pixel or a solid circle with a suitable radius randomly sampled from the background, and all clicks generated by the prompt simulator are input into the prompt encoder as interactive prompt input to generate a prompt embedding vector; when the prompt simulator is trained for the first time, a pixel or a solid circle with a suitable radius is randomly sampled from a true negative area covered by the ground truth mask and not covered by the predicted mask as a positive click, and a pixel or a solid circle with a suitable radius is randomly sampled from a false positive area not covered by the ground truth mask and covered by the predicted mask as a negative click; the prompt encoder obtains the prompt embedding vector through position encoding, and the dimension of the prompt vector is 256.

[0041] (5) In the training stage, the mask decoder is fine-tuned, the image embedding vector and the prompt embedding vector are input into the mask decoder, and the predicted mask is obtained.

[0042] The mask decoder in the embodiment includes prompt self-attention, prompt-image cross-attention, image vector upsampling, and multi-layer perception to obtain the probability of each pixel becoming a mask, and the predicted mask is obtained through threshold control; the structure of the image vector upsampling method is convolution transpose, LayerNorm, GELU activation function, convolution transpose, and GELU activation function; the structure of the multi-layer perception is a linear layer, a ReLU activation function, and a linear layer; the vector dimension of the mask decoder is equal to the dimension of the prompt embedding vector, which is 256; the depth of the two-way Transformer is 3, the dimension of the multi-layer perception is 1024, and the number of attention heads is 8.

[0043] (6) In the training stage, a suitable loss function (such as cross-entropy loss and focal loss) is used to calculate the loss between the predicted mask and the ground truth mask, to optimize the model and adjust the parameters that need to be fine-tuned; and then an optimizer (such as AdamW) is used to fine-tune the adaptive layer parameters of the image encoder, the prompt encoder parameters, and the mask decoder parameters to minimize the loss function and adjust the image encoder adaptive layer parameters, the prompt encoder parameters, and the mask decoder parameters.

[0044] The loss L is calculated by the following loss function formula in the embodiment:

[0045]

[0046]

[0047]

[0048] Wherein: S represents the predicted mask, G represents the ground truth mask, s i represents the confidence of the i-th pixel of the predicted mask, g i represents the confidence of the i-th pixel of the ground truth mask, γ is an adjustable factor; the optimizer uses AdamW, the initial learning rate is 1e-4, and the weight decay is 0.01.

[0049] (7) In the training process, the predicted mask generated by step (5) and the ground truth mask corresponding to the image are compared to obtain a comparison result, which is input into the hint simulator.

[0050] (8) In the training stage, the hint simulator obtains feedback to generate a positive click in the true negative area and a negative click in the false positive area. The simulated click map is obtained by accumulating all clicks of the image, which is used for the next training of the same image.

[0051] In the non-first training of the embodiment, the hint simulator randomly samples a pixel in the true negative area covered by the ground truth mask and not covered by the predicted mask as a positive click, and randomly samples a pixel in the false positive area not covered by the ground truth mask and covered by the predicted mask as a negative click; the hint encoder obtains the hint embedding vector by position encoding all clicks.

[0052] (9) In the training stage, steps (3) to (8) are repeated 3-5 times for the same image, and the training of all images in the training data set is considered as one iteration. The training is repeated until the predetermined number M is reached.

[0053] In the embodiment, the same image is repeatedly trained 3 times, and the predetermined number M is 100, that is, 100 epochs can obtain the expected fine-tuned model.

[0054] (10) In the inference stage, the user submits the picture that needs to be inferred, and the picture is input into the trained image encoder and adaptive layer to obtain the image embedding vector.

[0055] In the embodiment, the number of images used for inference and corresponding building labels is 200, and if the user submits a picture that exceeds the size required by the Transformer, the longest side length will be adjusted first.

[0056] (11) In the inference stage, the user manually left-clicks the target object that needs to be segmented on the image as a positive click, or right-clicks a pixel on the image that does not belong to this target object as a negative click; the click is input into the hint encoder to obtain the hint embedding vector.

[0057] (12) reasoning stage, the image embedding vector of the image encoder and the prompt embedding vector of the prompt encoder are input into the mask decoder to obtain a predicted mask, which is fed back to the user, and the user performs the next round of interaction according to the result, that is, steps (10) to (12) are repeated until the user expectation is reached.

[0058] If the input picture of the user is adjusted to meet the model requirements, the implementation mode will be restored by the bilinear interpolation method after the predicted segmentation result is output. Preliminary experiments show that for a remote sensing image with a resolution of 512*512, more than 6 clicks on average can achieve a reference performance of 85% mIoU, and more than 10 clicks on average can achieve a reference performance of 90% mIoU.

[0059] The above description of the embodiments is to facilitate the understanding and application of the present application by those skilled in the art. Those skilled in the art can easily make various modifications to the above embodiments, and apply the general principles described herein to other embodiments without creative labor. Therefore, the present application is not limited to the above embodiments, and any improvements and modifications of the present application made by those skilled in the art based on the disclosure of the present application shall be within the scope of protection of the present application.

Claims

1. An interactive segmentation method for remote sensing images based on fine-tuning of a large visual base model, comprising the following steps: (1) Obtain the visual basic model and its pre-trained model parameter file, and at the same time obtain the target remote sensing dataset and preprocess the remote sensing images therein. Specifically, use the pre-band converter to convert the RGB-IR four-band remote sensing image into an RGB three-channel remote sensing image through band selection. During the conversion process, first extract the RGB band to obtain a three-channel image, then extract the IR band to replace the R band in the original image to obtain another three-channel image. During training, the two three-channel images are sent to the training at the same time to ensure that all input image data are processed in the same format; The visual basic model consists of three parts: image encoder, hint encoder and mask decoder; (2) Fine-tune the image encoder structure by adding an adaptation layer to improve its feature extraction capability; During training, the pre-trained parameters of the image encoder are frozen, the parameters of the adaptation layer are trained using the target remote sensing dataset, and the pre-processed remote sensing image is input into the image encoder to obtain the image embedding vector; The image encoder uses a visual Transformer pre-trained by MAE, which consists of 12 ViT units in cascade, and each ViT unit adds an adaptation layer in parallel; each ViT unit is composed of LayerNorm, a multi-head attention mechanism layer, LayerNorm, and a multi-layer perceptron connected in sequence; the output of the adaptation layer is merged with the output of the multi-layer perceptron, and the adaptation layer contains two low-rank parameter matrices M down and M up , the input vector is first compared with the matrix M down The dot product is projected downward, and then activated by ReLU and the matrix M up The dot product is projected upward and finally multiplied by a scaling factor f scale Output; During training, freeze the image encoder pre-training parameters and only perform the matrix M down and M up Adjust the value of ; (3) Use the prompt simulator to generate positive clicks and negative clicks as interaction prompts and input them into the prompt encoder to obtain the prompt embedding vector; (4) Fine-tune the mask decoder structure and input the image embedding vector and the hint embedding vector into the mask decoder to obtain the predicted mask; The mask decoder is composed of a prompt self-attention mechanism layer, a prompt-image cross-attention mechanism layer, an image vector upsampling layer, and a multi-layer perceptron connected in sequence to obtain the probability of each pixel becoming a mask, and then obtain the predicted mask through threshold control; the image vector upsampling layer is composed of a transposed convolution layer, LayerNorm, GELU activation function, transposed convolution layer, and GELU activation function connected in sequence; (5) Design an appropriate loss function to train the large visual basic model; (6) The remote sensing image to be segmented and the user click interaction information are input into the trained model, and the corresponding prediction mask can be directly output as the image segmentation result.

2. The interactive remote sensing image segmentation method according to claim 1, wherein: The visual basic model obtained in step (1) is a Segment Anything Model and is pre-trained using the SA-1B dataset, wherein the image encoder is used to extract features from the image to obtain an embedding vector containing image feature information, the prompt encoder is used to perform prompt interactive processing on the image to obtain an embedding vector containing prompt feature information, and the mask decoder is used to perform mask processing on the above two sets of embedding vectors to obtain a predicted mask, thereby ultimately achieving interactive segmentation of the image.

3. The interactive remote sensing image segmentation method according to claim 1, wherein: The target remote sensing data set includes images to be annotated, mask images to be corrected, collected remote sensing images, and interaction point coordinates.

4. The interactive remote sensing image segmentation method according to claim 1, wherein: In the step (3), the prompt simulator is initialized during the first training. The prompt simulator will simulate user interaction to obtain positive clicks or negative clicks, that is, it simulates interactive sampling to sample a pixel or a solid circle of appropriate radius from the true value mask to generate a positive click, which is a click used to segment the target object mask; it simulates interactive sampling of a pixel or a solid circle of appropriate radius from the background to generate a negative click, which is a click used to remove the mask that does not belong to the target object; when it is not the first training, the prompt simulator will simulate interactive sampling of a pixel or a solid circle of appropriate radius as a positive click in the true negative area covered by the true value mask but not covered by the predicted mask, and simulate interactive sampling of a pixel or a solid circle of appropriate radius as a negative click in the false positive area not covered by the true value mask but covered by the predicted mask; the prompt simulator will input all the clicks it generates as interactive prompts into the prompt encoder to generate a prompt embedding vector through position encoding.

5. The interactive remote sensing image segmentation method according to claim 4, characterized in that: The simulated interactive sampling method is a sampling method designed by simulating the aggregation, significance, marginality and maximum discrimination strategies of manual clicks.

6. The interactive remote sensing image segmentation method according to claim 1, wherein: The expression of the loss function is as follows: Where: L is the total loss function, L CE is the cross entropy loss, L FL is the focusing loss, s i To predict the confidence of the i-th pixel in the mask, g i is the confidence of the i-th pixel in the true mask, γ is an adjustable factor, and N is the total number of pixels in the mask.

Images