Method and system for improving semantic segmentation performance of multi-source remote sensing image based on complementary prompt

Abstract

The invention provides a method and a system for improving semantic segmentation performance of a multi-source remote sensing image based on complementary prompt. According to the multi-modal remote sensing image semantic segmentation method, the high-level semantic features of the remote sensing images are extracted by using the basic model, and the middle features are mutually used as visual prompts during feature extraction, so that the complementary synergistic effect between the multi-modal remote sensing images is fully utilized, the multi-modal feature representation is perfected, and the multi-modal remote sensing image semantic segmentation performance is improved. According to the method, the cooperation and prompt effect between the multi-modal remote sensing images can be fully utilized, and the problem that the existing method excessively pursues common feature expression between modals is better solved. According to the method provided by the invention, middle features can be mutually utilized as visual prompts during extraction of other modalities in the modal feature extraction process, so that the model is more focused on the missing feature part of the current modal, more comprehensive multi-modal feature representation is learned, and the method is applied to subsequent downstream tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a method for semantic segmentation of multimodal remote sensing images, and in particular to a method and system for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues. Background Art

[0002] In recent years, the diversification of imaging methods has led to the diversification of remote sensing images. For example, satellites such as Gaofen-2, WorldView-2, and Sentinel-2 can obtain optical images, and satellites such as Gaofen-3, TerraSAR-X, and Sentinel-1 can obtain synthetic aperture radar (SAR) images. Although remote sensing images of different modalities provide a more comprehensive description of earth observation, their heterogeneity also brings new challenges to remote sensing image interpretation. With the development of deep learning technology, multimodal remote sensing basic models have performed well in many downstream tasks in the remote sensing field with massive training data and a large number of model parameters. However, these basic model methods overly pursue the unified representation of cross-modal image features, and often ignore the information exchange and prompting role between different modal features. Some studies have strengthened the feature expression of each modality to a certain extent by decoupling the common and unique parts of multimodal data features, but they still have not gotten rid of the limitations of modality-specific feature extractors, or have achieved multimodal feature fusion through a unified encoder, but its performance will drop sharply due to the heterogeneity of the data. Summary of the invention

[0003] In view of the defects in the prior art, the present invention draws on the strategy of the human brain to process external visual information, proposes a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, which uses the feature information extracted from different modal images as supplementary cues when extracting features from other modalities, fully utilizes the diversity of multimodal image information, and promotes the generation of a more comprehensive multimodal unified representation, which has not been achieved yet.

[0004] The method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues provided by the present invention comprises the following specific steps:

[0005] Step 1: Input the labeled and spatially aligned Modal1 and Modal2 images into the dual-branch feature encoder at the same time;

[0006] Step 2: The multimodal data first passes through the image block embedding layer corresponding to each modality, converting the image data into a sequence, expressed as The superscripts indicate the layers in the dual-branch feature encoder;

[0007] Step 3: Multimodal data features Input to the Transformer Block corresponding to each mode to get the output of this layer of Transformer Block

[0008] Step 4: Multimodal features obtained in step 2 The two are simultaneously input into the complementary cue generator to generate complementary visual cues C1 and V2;

[0009] Step 5: Add the output of each Transformer Block to the corresponding complementary prompt data prompt to obtain the input of the next Transformer Block Recursively, the input of each subsequent Transformer Block layer is equal to the output of the previous Transformer Block plus the complementary visual cue;

[0010] Step 6: Output of the last Transformer Block At the same time, they are input into the modal feature fusion module to obtain the final output F of the dual-branch feature encoder as a multi-modal feature representation;

[0011] Step 7: The multimodal feature representation is input into the decoder to obtain the category prediction of each pixel, which is the semantic segmentation result.

[0012] Furthermore, in step 4, the specific processing process of the complementary prompt generator is: using multimodal features One of them is the main mode, and the other is the prompt module. and The differential features between the modalities are obtained by subtraction, and the channel attention mechanism is used to perform global average pooling on the differential features to obtain the channel weights, which are weighted with the corresponding multimodal features, and then passed through a multi-layer perceptron to generate a complementary visual cue V1 for the main modality; the main modality and the cue modality are switched to obtain another complementary visual cue V2.

[0013] Furthermore, the complementary cue generator is placed in parallel in each layer of the dual-branch feature encoder.

[0014] Furthermore, the processing of the modal feature fusion module is as follows;

[0015] Will The elements are added one by one, and then the global average pooling is performed, and then the input is input to the dual-branch multi-layer perceptron layer. The fusion ratio of different modal image features is adaptively determined through the activation function, and combined with The output F is obtained by weighting.

[0016] Furthermore, when training the dual-branch feature encoder and decoder, the parameters of the image block embedding layer and TransformerBlock are not updated with gradients.

[0017] Furthermore, when training the dual-branch feature encoder and decoder, the cross entropy loss function is used Train and finally back-propagate to the parameters that need gradient update, where one_hot is the only hot encoding, y i is the category prediction result of the i-th pixel in step 7, p i is the true value of the corresponding pixel, and N is the number of pixels.

[0018] Furthermore, the processing of Transformer Block includes two parts. The first part is to apply layer normalization to the input sequence and then input it into the self-attention module, using the attention mechanism to consider the global relationship in the input sequence. The second part is to apply layer normalization and then input it into the multi-layer perceptron for forward propagation. Both parts use residual connections to improve the training stability of the model.

[0019] Furthermore, the method also includes step 8, which uses two indicators, mean intersection over union (mIoU) and mean pixel accuracy (mPA), to evaluate the performance of semantic segmentation results on a multimodal remote sensing dataset.

[0020] The present invention also provides a system for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, comprising a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as described in the above scheme.

[0021] The principle of the present invention is to utilize the ability of the basic model to extract high-level semantic features of remote sensing images, and to utilize intermediate features as visual cues for feature extraction, so as to fully utilize the complementary synergy between multimodal remote sensing images, improve multimodal feature representation, and improve the semantic segmentation performance of multimodal remote sensing images.

[0022] The purpose of the present invention is to provide a method for improving the semantic segmentation performance of multimodal remote sensing images based on complementary visual cues, which can fully utilize the synergy and cueing effects between multimodal remote sensing images, and better solve the problem that the existing methods are too focused on the expression of common features between modalities. The proposed method can mutually utilize intermediate features as visual cues for other modal extractions during the process of modal feature extraction, so that the model can focus more on the missing feature parts of the current modality, learn a more comprehensive and integrated multimodal feature representation, and apply it to subsequent downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1This is a diagram of a multimodal remote sensing image segmentation network model based on complementary visual cues in an embodiment of the present invention.

[0024] Figure 2 4 is a network model diagram of a complementary prompt generator in an embodiment of the present invention.

[0025] Figure 3 4 is a network model diagram of the modal feature fusion module in an embodiment of the present invention.

[0026] Figure 4 2 is a network model diagram of the Transformer encoder structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings.

[0028] The present invention provides a method for improving the semantic segmentation performance of multimodal remote sensing images based on complementary visual cues. The method is inspired by the human brain's strategy for processing external visual information, and uses the knowledge learned from different modal images as cues and guidance in the learning process of other modalities, so as to promote the acquisition of unified representation of multimodal remote sensing image features, thereby improving its semantic segmentation performance. The network includes two fully structured feature extractors, which are respectively used to extract features of different modal images, and their parameters are fine-tuned according to the corresponding modalities and kept frozen in the subsequent training process. The network simultaneously inputs multimodal remote sensing image data, on the one hand, propagates backward along the corresponding feature extractor, and on the other hand, simultaneously inputs into the complementary cue generation module, and emphasizes the difference between the main modal features and the additional modal features through the channel attention mechanism, so that the visual cues focus more on the feature information lacking in the main modal image, and better promote information exchange between different modalities. Finally, the different modal features obtained by the dual-branch feature extractor are input into the inter-modal fusion module, and the fusion proportion of the different modal image features is adaptively determined according to the similarity of the inter-modal features and the importance of the modal information, so as to achieve effective fusion of multimodal remote sensing image features. The method for improving the semantic segmentation performance of multimodal remote sensing images based on complementary visual cues adopted in the present invention can effectively realize information exchange in the process of learning unified representation of multimodal features, make full use of the complementary effect of multimodal information, and achieve more accurate semantic segmentation of multimodal remote sensing images.

[0029] like Figure 1 As shown, an embodiment of the present invention provides a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, comprising the following steps:

[0030] Step 1: Input the labeled and spatially aligned Modal1 and Modal2 images into the dual-branch feature encoder at the same time, where Modal1 represents the image of Modal 1 and Modal2 represents the image of Modal 2.

[0031] Step 2: The multimodal data first passes through the image block embedding layer corresponding to each modality, namely the PatchEmbedding layer, which converts the image data into a sequence for subsequent processing by the Transformer layer, expressed as The superscripts indicate the layers in the encoder.

[0032] Step 3: Multimodal Data Features Input to the corresponding mode Figure 4 In the TransformerBlock shown, the parameters of the Transformer Block are no longer updated with gradients. The data features undergo a series of self-attention and forward propagation operations to obtain the output of this layer of Transformer Block.

[0033] The processing of Transformer Block includes two parts. The first part is to apply layer normalization to the input sequence and then input it into the self-attention module, using the attention mechanism to consider the global relationship in the input sequence. The second part is to apply layer normalization and then input it into the multi-layer perceptron for forward propagation. Both parts use residual connections to improve the training stability of the model.

[0034] Step 4: Figure 2 As shown, the multimodal features obtained in step 2 At the same time, input them into the complementary prompt generator, that is, the complementary prompt generator, and make a difference. The differentiated features are obtained, and the channel attention mechanism is used to perform global average pooling on the differentiated features to obtain channel weights, emphasizing the missing features of the modality, so as to generate complementary visual prompts V1 for the other modality. Two sets of Complementary Prompt Generators are located in parallel in each layer of the encoder, and the main modality is switched to obtain another complementary visual prompt V2.

[0035] Step 5: Add the output of each Transformer Block to the corresponding complementary data prompt to get the input of the next Transformer Block Recursively following the above steps, the input of each subsequent Transformer Block layer is equal to the output of the previous Transformer Block layer plus the visual cue.

[0036] Step 6: Figure 3 As shown, the output of the last layer of Transformer Block At the same time, they are input into the Inter-modal Fusion Module, and according to the similarity of inter-modal features and the importance of modal information, The images are added together and globally averaged pooled. The fusion weights of the image features of different modalities are adaptively determined through the fully connected layer and the softmax layer. The final output F of the encoder is obtained after addition as a multimodal feature representation.

[0037] Step 7: The multimodal feature representation is input into the decoder head to obtain the category prediction y for each pixel i , and the true value p of the corresponding pixel i Compare, that is, compare with Ground Truth and calculate the loss function Among them, one_hot is a one-hot encoding, and is back-propagated to the parameters that need to be updated by gradients.

[0038] The performance of the present invention was tested on the multimodal remote sensing dataset WHU-OPT-SAR, and the test results are as follows:

[0039]

[0040] The present invention is superior to the multimodal semantic segmentation algorithms proposed in recent years in terms of two indicators for measuring the performance of semantic segmentation algorithms, namely, mean intersection over union (mIoU) and mean pixel accuracy (mPA).

[0041] On the other hand, an embodiment of the present invention also provides a system for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as described in the above scheme.

[0042] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, characterized in that: The specific steps include: Step 1: Input the labeled and spatially aligned Modal1 and Modal2 images into the dual-branch feature encoder at the same time; Step 2: The multimodal data first passes through the image block embedding layer corresponding to each modality, converting the image data into a sequence, expressed as The superscripts indicate the layers in the dual-branch feature encoder; Step 3: Multimodal data features Input to the Transformer Block corresponding to each mode to get the output of this layer of Transformer Block Step 4: Multimodal features obtained in step 2 The two are simultaneously input into the complementary cue generator to generate complementary visual cues V1 and V2; Step 5: Add the output of each Transformer Block to the corresponding complementary prompt data prompt to obtain the input of the next Transformer Block Recursively, the input of each subsequent Transformer Block layer is equal to the output of the previous Transformer Block plus the complementary visual cue; Step 6: Output of the last Transformer Block At the same time, they are input into the modal feature fusion module to obtain the final output F of the dual-branch feature encoder as a multi-modal feature representation; Step 7: The multimodal feature representation is input into the decoder to obtain the category prediction of each pixel, which is the semantic segmentation result.

2. A method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as claimed in claim 1, characterized in that: In step 4, the specific processing process of the complementary prompt generator is: using multimodal features One of them is the main mode, and the other is the prompt module. and The differential features between the modalities are obtained by subtraction, and the differential features are globally averaged pooled using the channel attention mechanism to obtain the channel weights. The channel weights are weighted with the corresponding multimodal features, and then passed through a multi-layer perceptron to generate a complementary visual cue C1 for the main modality; the main modality and the cue modality are switched to obtain another complementary visual cue C2.

3. A method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as claimed in claim 1, characterized in that: The complementary cue generators are placed in parallel in each layer of the dual-branch feature encoder.

4. A method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as claimed in claim 1, characterized in that: The processing process of the modal feature fusion module is as follows; Will The elements are added one by one, and then the global average pooling is performed, and then the input is input to the dual-branch multi-layer perceptron layer. The fusion ratio of different modal image features is adaptively determined through the activation function, and combined with The output F is obtained by weighting.

5. The method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as claimed in claim 1, characterized in that: When training the dual-branch feature encoder and decoder, the parameters of the image block embedding layer and the Transformer Block are not updated with gradients.

6. A method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as claimed in claim 1, characterized in that: When training the dual-branch feature encoder and decoder, the cross entropy loss function is used Train and finally back-propagate to the parameters that need gradient update, where one_hot is the only hot encoding, y i is the category prediction result of the i-th pixel in step 7, p i is the true value of the corresponding pixel, and N is the number of pixels.

7. The method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as claimed in claim 1, characterized in that: The processing of Transformer Block includes two parts. The first part is to apply layer normalization to the input sequence and then input it into the self-attention module, using the attention mechanism to consider the global relationship in the input sequence. The second part is to apply layer normalization and then input it into the multi-layer perceptron for forward propagation. Both parts use residual connections to improve the training stability of the model.

8. The method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as claimed in claim 1, characterized in that: The method also includes step 8, which uses the two indicators of mean intersection over union (mIoU) and mean pixel accuracy (mPA) to evaluate the performance of semantic segmentation results on multimodal remote sensing datasets.

9. A system for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as described in any one of claims 1 to 8.

Images