A method and system for improving the performance of multi-source remote sensing image semantic segmentation based on complementary cues

Abstract

The application provides a method and system for improving the semantic segmentation performance of multi-source remote sensing images based on complementary prompts. The application utilizes the ability of the base model to extract high-level semantic features of remote sensing images, and uses intermediate features as visual prompts during feature extraction. This makes full use of the complementary and collaborative effects between multi-modal remote sensing images, improves multi-modal feature representation, and improves the semantic segmentation performance of multi-modal remote sensing images. This method can fully utilize the collaboration and prompting effect between multi-modal remote sensing images, and better solve the problem that existing methods excessively pursue common feature expression between modalities. The proposed method can utilize intermediate features as visual prompts during feature extraction of other modalities, so that the model focuses more on the missing feature part of the current modality, learns more comprehensive multi-modal feature representation, and is applied to subsequent downstream tasks.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and relates to a method for semantic segmentation of multimodal remote sensing images, and particularly to a method and system for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues. Background Technology

[0002] In recent years, the diversification of imaging methods has led to the diversification of remote sensing images. For example, satellites such as Gaofen-2, WorldView-2, and Sentinel-2 can acquire optical images, while satellites such as Gaofen-3, TerraSAR-X, and Sentinel-1 can acquire synthetic aperture radar (SAR) images. Although remote sensing images of different modalities provide a more comprehensive description of Earth observation, their heterogeneity also brings new challenges to remote sensing image interpretation. With the development of deep learning technology, multimodal remote sensing basic models have performed well in many downstream tasks in the field of remote sensing, thanks to massive training data and a large number of model parameters. However, these basic model methods overemphasize the unified representation of cross-modal image features, often neglecting the information exchange and cueing role between different modal features. Some studies have strengthened the feature representation of each modality to some extent by decoupling the common and unique parts of multimodal data features, but they still have not escaped the limitations of modality-specific feature extractors. Alternatively, they have achieved the fusion of multimodal features through a unified encoder, but its performance drops sharply due to the heterogeneity of the data. Summary of the Invention

[0003] To address the shortcomings of existing technologies, and drawing on the strategies employed by the human brain in processing external visual information, this invention proposes a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues. This method uses feature information extracted from different modalities as supplementary cues when extracting features from other modalities, fully utilizing the diversity of multimodal image information to promote the generation of a more comprehensive unified multimodal representation. This approach has not yet been implemented by anyone before.

[0004] The method for improving semantic segmentation performance of multi-source remote sensing images based on complementary cues provided by this invention includes the following specific steps:

[0005] Step 1: Input the labeled and spatially aligned images of Modal1 and Modal2 into the dual-branch feature encoder simultaneously;

[0006] Step 2: The multimodal data first passes through the image patch embedding layer corresponding to each modality, converting the image-format data into a sequence, which is then represented as... The superscript indicates the layer number in the dual-branch feature encoder;

[0007] Step 3, Multimodal Data Features The inputs are fed into the Transformer Block corresponding to their respective modes to obtain the output of that Transformer Block.

[0008] Step 4, the multimodal features obtained in Step 2 Simultaneously input into the complementary cue generator to generate complementary visual cues C1 and V2;

[0009] Step 5: Add the output of each Transformer Block to the corresponding complementary cue data to obtain the input of the next Transformer Block. Following this logic, the input to each subsequent Transformer Block layer is equal to the output of the previous Transformer Block layer plus complementary visual cues;

[0010] Step 6, Output of the last Transformer Block Simultaneously, the input is fed into the modal feature fusion module to obtain the final output F of the dual-branch feature encoder, which serves as the multimodal feature representation;

[0011] Step 7: The multimodal feature representation is input into the decoder to obtain the category prediction for each pixel, which is the semantic segmentation result.

[0012] Furthermore, in step 4, the specific processing procedure of the complementary cue generator is as follows: using multimodal features One is the primary modality, and the other serves as a prompting module, through... and The difference between modalities is obtained by subtraction, and the differential features are globally averaged and pooled using the channel attention mechanism to obtain channel weights. The channel weights are then weighted with the corresponding multimodal features and passed through a multilayer perceptron to generate a complementary visual cue V1 for the main modality. The main modality and the cue modality are switched to obtain another complementary visual cue V2.

[0013] Furthermore, complementary cue generators are located side-by-side in each layer of the dual-branch feature encoder.

[0014] Furthermore, the processing procedure of the modal feature fusion module is as follows;

[0015] Will Element-wise summation is performed, followed by global average pooling, and then the result is fed into a two-branch multilayer perceptron layer. The activation function adaptively determines the fusion weight of different modalities of image features and integrates them with the data. The weighted average is used to obtain the output F.

[0016] Furthermore, when training the dual-branch feature encoder and decoder, the parameters of the image patch embedding layer and TransformerBlock are not updated with gradients.

[0017] Furthermore, when training the dual-branch feature encoder and decoder, the cross-entropy loss function is used. Training is performed, and finally backpropagation is used to update the parameters that require gradient updates, where one-hot encoding is used, and y i For the category prediction result of the i-th pixel in step 7, p i is the true value of the corresponding pixel, and N is the number of pixels.

[0018] Furthermore, the Transformer Block's processing consists of two parts. The first part is to apply layer standardization to the input sequence and then input it into the self-attention module, using the attention mechanism to consider the global relationships in the input sequence. The second part is to apply layer standardization to the multilayer perceptron for forward propagation. Both parts use residual connections to improve the training stability of the model.

[0019] Furthermore, step 8 is included, which uses the average intersection-union ratio (mIoU) and average pixel accuracy (mPA) as two metrics to evaluate the performance of semantic segmentation results on a multimodal remote sensing dataset.

[0020] The present invention also provides a system for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, comprising a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as described above.

[0021] The principle of this invention is to utilize the ability of a basic model to extract high-level semantic features from remote sensing images, and to use intermediate features as visual cues during feature extraction. This fully leverages the complementary and synergistic effects between multimodal remote sensing images, improves multimodal feature representation, and enhances the semantic segmentation performance of multimodal remote sensing images.

[0022] The purpose of this invention is to provide a method for improving the semantic segmentation performance of multimodal remote sensing images based on complementary visual cues. This method can fully utilize the synergistic and cueing effects between multimodal remote sensing images, better addressing the problem of existing methods overemphasizing shared features between modalities. The proposed method can mutually utilize intermediate features as visual cues for other modalities during feature extraction, allowing the model to focus more on the missing features of the current modality, learning a more comprehensive and integrated multimodal feature representation for application in subsequent downstream tasks. Attached Figure Description

[0023] Figure 1This is a diagram of a multimodal remote sensing image segmentation network model based on complementary visual cues in an embodiment of the present invention.

[0024] Figure 2 This is a network model diagram of the complementary hint generator in an embodiment of the present invention.

[0025] Figure 3 This is a network model diagram of the modal feature fusion module in an embodiment of the present invention.

[0026] Figure 4 This is a network model diagram of the Transformer encoder structure in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings.

[0028] This invention provides a method for improving the semantic segmentation performance of multimodal remote sensing images based on complementary visual cues. Inspired by the human brain's strategies for processing external visual information, this method uses knowledge learned from images of different modalities as cues and guidance in the learning process of other modalities, promoting the acquisition of unified representations of multimodal remote sensing image features, thereby improving its semantic segmentation performance. The network contains two fully structured feature extractors, each used to extract features from different modalities. Their parameters are fine-tuned according to the corresponding modality and remain frozen during subsequent training. The network simultaneously inputs multimodal remote sensing image data, propagating backward along the corresponding feature extractors and simultaneously inputting it into the complementary cue generation module. Through a channel attention mechanism, the differences between the main modality features and the additional modality features are emphasized, allowing the visual cues to focus more on the feature information missing from the main modality image, thus better promoting information exchange between different modalities. Finally, the different modal features obtained by the dual-branch feature extractors are input into the intermodality fusion module. Based on the similarity of the intermodal features and the importance of the modal information, the fusion weight of the different modal image features is adaptively determined, achieving effective fusion of multimodal remote sensing image features. The method for improving the semantic segmentation performance of multimodal remote sensing images based on complementary visual cues adopted in this invention can effectively realize information exchange in the process of learning unified representation of multimodal features, make full use of the complementary effect of multimodal information, and achieve higher-precision semantic segmentation of multimodal remote sensing images.

[0029] like Figure 1 As shown, this embodiment of the invention provides a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, comprising the following steps:

[0030] Step 1: Input the labeled and spatially aligned images of Modal1 and Modal2 into the dual-branch feature encoder simultaneously. Modal1 represents the image of Modal 1, and Modal2 represents the image of Modal 2.

[0031] Step 2: The multimodal data first passes through the respective patch embedding layer (PatchEmbedding layer) of its modality, converting the image data into a sequence to facilitate processing by the subsequent Transformer layer. The superscript indicates the layer number of the encoder.

[0032] Step 3: Multimodal data features Input to the respective modal corresponding to, such as Figure 4 In the TransformerBlock shown, the parameters of the Transformer Block are no longer updated with gradients. The data features undergo a series of self-attention and forward propagation operations to obtain the output of this Transformer Block.

[0033] The Transformer Block's processing consists of two parts. The first part applies layer standardization to the input sequence and then inputs it into the self-attention module, using the attention mechanism to consider the global relationships in the input sequence. The second part applies layer standardization and then inputs it into the multilayer perceptron for forward propagation. Both parts use residual connections to improve the training stability of the model.

[0034] Step 4: As Figure 2 As shown, the multimodal features obtained in step 2 Simultaneously, the input is fed into the Complementary Prompt Generator, and the difference is calculated... Differential features are obtained, and channel attention mechanism is used to perform global average pooling on the differential features to obtain channel weights, emphasizing the missing features of this modality, thereby generating complementary visual cues V1 for another modality. Two sets of complementary prompt generators are located side by side in each layer of the encoder, and switching the main modality obtains another complementary visual cue V2.

[0035] Step 5: Add the output of each Transformer Block to the corresponding complementary data cue to obtain the input of the next Transformer Block. Following the steps described above, the input to each subsequent Transformer Block layer is equal to the output of the previous Transformer Block layer plus a visual cue.

[0036] Step 6: As Figure 3 As shown, the output of the last Transformer Block Simultaneously, the data is input into the Inter-modal Fusion Module, where, based on the similarity of features between modalities and the importance of modal information, the data is fused. The features are summed and global average pooling is performed. The fusion weight of different modal image features is adaptively determined through fully connected layers and softmax layers. The summation yields the final output F of the encoder, which serves as a representation of the multimodal features.

[0037] Step 7: The multimodal feature representation is input into the decoder (Decoder Head) to obtain the class prediction y for each pixel. i , and the true value p of the corresponding pixel i The comparison involves comparing the data with the Ground Truth and calculating the loss function. One-hot encoding is used, and it is backpropagated to the parameters that need to be updated with gradients.

[0038] The performance of this invention was tested on the multimodal remote sensing dataset WHU-OPT-SAR. The test results are as follows:

[0039]

[0040] This invention outperforms recent multimodal semantic segmentation algorithms in both average intersection-union ratio (mIoU) and average pixel accuracy (mPA), two metrics that measure the performance of semantic segmentation algorithms.

[0041] In another aspect, embodiments of the present invention also provide a system for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as described above.

[0042] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, characterized in that, The specific steps include the following: Step 1: Input the labeled and spatially aligned images of Modal1 and Modal2 into the dual-branch feature encoder simultaneously; Step 2: The multimodal data first passes through the image patch embedding layer corresponding to each modality, converting the image-format data into a sequence, which is then represented as... , The superscript indicates the layer number in the dual-branch feature encoder; Step 3, Multimodal Data Features , The inputs are fed into the Transformer Block corresponding to their respective modes to obtain the output of that Transformer Block. , ; Step 4, the multimodal features obtained in Step 2 , Simultaneously input into the complementary cue generator to generate complementary visual cues. and ; In step 4, complementary cue generators are located side-by-side in each layer of the dual-branch feature encoder. The specific processing procedure of the complementary cue generator is as follows: using multimodal features... , One is the primary modality, and the other serves as a prompting module, through... and The difference between modalities is obtained by subtraction, and the differential features are then subjected to global average pooling using a channel attention mechanism to obtain channel weights. These channel weights are then weighted with the corresponding multimodal features and passed through a multilayer perceptron to generate complementary visual cues for the dominant modality. ; Switch between the primary modality and the cue modality to obtain another complementary visual cue. ; Step 5: Add the output of each Transformer Block to the corresponding complementary cue data to obtain the input of the next Transformer Block. , Recursively, the input of each subsequent Transformer Block layer is equal to the output of the previous Transformer Block plus complementary visual cues; Step 6, Output of the last Transformer Block , Simultaneously, the input is fed into the modal feature fusion module to obtain the final output of the dual-branch feature encoder. , as a multimodal feature representation; Step 7: The multimodal feature representation is input into the decoder to obtain the category prediction for each pixel, which is the semantic segmentation result.

2. The method for improving semantic segmentation performance of multi-source remote sensing images based on complementary cues as described in claim 1, characterized in that: The processing procedure of the modal feature fusion module is as follows; Will , Element-wise summation is performed, followed by global average pooling, and then the result is fed into a two-branch multilayer perceptron layer. The activation function adaptively determines the fusion weight of different modalities of image features and... , Weighted summation yields the output. .

3. The method for improving semantic segmentation performance of multi-source remote sensing images based on complementary cues as described in claim 1, characterized in that: When training the dual-branch feature encoder and decoder, the parameters of the image patch embedding layer and the Transformer Block are not updated with gradients.

4. The method for improving semantic segmentation performance of multi-source remote sensing images based on complementary cues as described in claim 1, characterized in that: When training the dual-branch feature encoder and decoder, the cross-entropy loss function is used. Training is performed, and finally backpropagation is used to update the parameters that require gradients, where one-hot encoding is used. The category prediction result for the i-th pixel in step 7. is the true value of the corresponding pixel, and N is the number of pixels.

5. The method for improving semantic segmentation performance of multi-source remote sensing images based on complementary cues as described in claim 1, characterized in that: The Transformer Block's processing consists of two parts. The first part applies layer standardization to the input sequence and then inputs it into the self-attention module, using the attention mechanism to consider the global relationships in the input sequence. The second part applies layer standardization and then inputs it into the multilayer perceptron for forward propagation. Both parts use residual connections to improve the training stability of the model.

6. The method for improving semantic segmentation performance of multi-source remote sensing images based on complementary cues as described in claim 1, characterized in that: It also includes step 8, which uses the average intersection-union ratio (mIoU) and average pixel accuracy (mPA) as two metrics to evaluate the performance of semantic segmentation results on a multimodal remote sensing dataset.

7. A system for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a method for improving the semantic segmentation performance of multi-source remote sensing images based on complementary cues as described in any one of claims 1-6.

Images