Remote sensing semantic segmentation method based on vision-language model encoder and double-branch weighted decoder

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a remote sensing semantic segmentation method based on a vision-language model encoder and a double-branch decoder, which comprises the following steps: firstly, collecting public remote sensing image data, constructing a large-scale image-text data set, and pre-training an image encoder and a text encoder in a vision-language model; thirdly, designing a remote sensing semantic segmentation framework of an encoder-decoder, which comprises a double-branch weighted encoder structure and is equipped with a multi-scale feature fusion decoder and a feature difference aggregation decoder; in the third step, a vision-language model image encoder obtained through pre-training is migrated into the segmentation framework to serve as an image feature extractor. Then, training the whole framework through a large-scale public remote sensing semantic segmentation data set, and independently learning the double-branch decoders by using a cross entropy loss function; in order to improve the segmentation precision, the performance of the double-branch decoder is optimized through weight constraint. And finally, saving a segmentation framework weight obtained by training, and loading the weight in a reasoning stage to generate a visual segmentation result. According to the method, the vision-language pre-training model and the double-branch decoder structure are effectively combined, so that the precision of a remote sensing image segmentation task is improved.

Description

Technical Field

[0001] This invention relates to remote sensing semantic segmentation methods, and particularly to remote sensing semantic segmentation methods based on a vision-language model, belonging to the field of computer technology. Background Technology

[0002] With the rapid development of remote sensing sensors and satellites, acquiring high-resolution remote sensing images has become much easier. Semantic segmentation of remote sensing images has been applied in geographic information applications, including urban planning, land resource management, and disaster assessment. Traditional methods for semantic segmentation of remote sensing images mainly rely on manually designed feature extractors and classifiers, which leads to a lack of generalization ability across different remote sensing scenarios. Some classic segmentation models based on convolutional neural networks (CNNs) have achieved significant success in remote sensing image segmentation tasks. However, most of these works only use deep features for prediction, ignoring scale variations. In addition, some studies have shown that simple fusion strategies may also lead to redundant feature information.

[0003] In recent years, visual language models have received widespread attention in the field of computer vision. These visual language models can acquire general knowledge and reduce dependence on specific data by pre-training on large-scale image-text datasets. However, due to the domain differences in image distribution, directly applying general visual language models to downstream remote sensing tasks may lead to performance degradation. Some researchers have fine-tuned visual-language models on small-scale remote sensing image-text datasets, demonstrating excellent generalization performance across a range of downstream tasks. Therefore, this invention designs a remote sensing semantic segmentation framework based on a visual-language model encoder-decoder structure. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder, which provides an effective solution for the application of visual-language models in remote sensing semantic segmentation tasks.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] The remote sensing semantic segmentation method based on a vision-language model encoder and a dual-branch weighted decoder includes the following steps:

[0007] Step 1: Collect publicly available remote sensing image data, construct a large-scale remote sensing image-text dataset, and pre-train the image encoder and text encoder of the vision-language model;

[0008] Step 2: Construct an encoder-decoder remote sensing semantic segmentation framework and design a dual-branch weighted encoder structure, which includes a multi-scale feature fusion decoder and a feature difference aggregation decoder.

[0009] Step 3: Transfer the image encoder of the pre-trained visual-language model obtained in Step 1 to the remote sensing semantic segmentation framework in Step 2 as the encoder to extract image features;

[0010] Step 4: Train the entire remote sensing semantic segmentation framework using a large-scale public remote sensing semantic segmentation dataset. The dual-branch decoder is learned independently using two loss functions, each with cross-entropy loss.

[0011] Step 5: Set up dual-branch decoders with different weight constraints to improve the segmentation accuracy of the entire segmentation framework;

[0012] Step 6: Save the weights of the remote sensing semantic segmentation framework trained in Step 5. During the inference phase, load the weight parameters to obtain the visualized segmentation results.

[0013] As a preferred embodiment of the present invention, the specific process of step 1 is as follows:

[0014] 1.1 Collect publicly available remote sensing image data from multiple sources. These images may cover different resolutions, different scenes (such as cities, forests, farmland, etc.), and remote sensing images acquired at different altitudes. This diverse data source helps the model adapt to diverse remote sensing scenes, thereby improving its generalization ability across various scenarios. For each sample image X, a corresponding image text description Y is generated using a multimodal large language model (such as CLIP, BLIP, etc.). caption These text descriptions provide a semantic global overview of each image, describing its content such as terrain features, environmental conditions, and building types. Next, a pre-trained image-text comparison model is used to calculate the sample image X and its corresponding text description Y. caption The similarity score is used to evaluate the consistency between the text description and the image content. Based on the similarity score, high-quality image-text pairs are selected to ensure that the descriptions accurately reflect the key information of the image.

[0015] 1.2 After selecting high-quality image-text pairs, the model is further fine-tuned based on this data. Through fine-tuning of the visual encoder and text encoder using the InfoNCE contrastive loss function, the model will gradually improve its ability to capture remote sensing image features. The formula for calculating the contrastive loss function is as follows:

[0016]

[0017] Where N is the batch size, sim represents the similarity function, and τ is the learnable temperature parameter.

[0018] As a preferred embodiment of the present invention, the specific process of step 2 is as follows:

[0019] 2.1 A remote sensing semantic segmentation framework based on an encoder-decoder architecture is constructed, mainly consisting of an image encoder and a dual-branch weighted decoder structure. The image encoder structure is identical to that in the vision-language model. The dual-branch weighted decoder structure includes a multi-scale feature fusion decoder and a feature difference aggregation decoder. The multi-scale feature fusion decoder integrates features at different scales to reduce the loss of semantic information in multi-scale features. The feature difference aggregation decoder, as an auxiliary branch, captures the differences between features at different levels, thereby enhancing feature complementarity.

[0020] 2.2 Multi-scale Feature Fusion Decoder performs multi-scale fusion through simple convolution and upsampling to obtain multi-scale features. The specific calculation formula is as follows:

[0021]

[0022] Where up represents the upsampling operation and conv represents the convolution function.

[0023] 2.3 The feature difference aggregation decoder captures the differences between multi-scale feature maps by subtracting feature differences, and the resulting difference features are: The specific calculation formula is as follows:

[0024]

[0025] Where || represents absolute value operation, up represents upsampling by 2x through bilinear interpolation, conv represents convolution function, and the convolution kernel k×k={1×1,3×3,5×5}.

[0026] As a preferred embodiment of the present invention, the specific process described in step 3 is as follows:

[0027] The image encoder of the pre-trained visual-language model obtained in step 1 is transferred to the remote sensing semantic segmentation framework in step 2 as an encoder to extract image features. This implicitly utilizes the rich remote sensing prior knowledge in the pre-trained visual-language model, providing a good starting point for further training on the downstream remote sensing image semantic segmentation dataset.

[0028] As a preferred embodiment of the present invention, the specific process of step 4 is as follows:

[0029] Cross-entropy loss is used as the loss function to independently learn the dual-branch decoder. The specific calculation formula is as follows:

[0030]

[0031] Where L1 and L2 represent the loss functions of the multi-scale fusion decoder and the feature difference aggregation decoder, respectively. i This represents the true label of the i-th pixel. and represents the predictions of the multi-scale fusion decoder and the feature difference aggregation decoder for the i-th pixel, respectively, and N represents the number of pixels in the training sample image.

[0032] As a preferred embodiment of the present invention, the specific process of step 5 is as follows:

[0033] A dual-branch decoder with different weights is configured to balance the impact of each branch decoder on the final prediction result. The total loss of the entire segmentation framework is composed of their weighted sum. The specific calculation formula is as follows:

[0034] L total =αL1+(1-ɑ)L2

[0035] Where L1 and L2 are the losses of the dual-branch decoder in step 4, and α and 1-α are their respective weights.

[0036] As a preferred embodiment of the present invention, the specific process of step 6 is as follows:

[0037] Save the weights of the remote sensing semantic segmentation framework trained in step 5. In the inference stage, input the test sample image into the segmentation framework to obtain the predicted segmentation result. Finally, visualize the predicted segmentation mask as the final output of this invention.

[0038] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0039] 1. This invention proposes a remote sensing semantic segmentation framework based on a vision-language model, which transfers the image encoder weights of the vision-language model to the segmentation framework, implicitly utilizing the rich remote sensing prior knowledge in the pre-trained vision-language model.

[0040] 2. This invention proposes a dual-branch weighted decoder structure. The multi-scale feature fusion decoder integrates features at different scales to reduce the loss of semantic information in multi-scale features. The feature difference aggregation decoder serves as an auxiliary branch to capture the differences between features at different levels, thereby enhancing feature complementarity. Different weights are set to constrain the dual-branch decoder, balancing the impact of different branch decoders on the final prediction result and improving segmentation accuracy. Attached Figure Description

[0041] Figure 1 This is an algorithm flowchart of the remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder, as presented in this invention.

[0042] Figure 2 This is a schematic diagram of the remote sensing semantic segmentation method framework based on a vision-language model encoder and a dual-branch weighted decoder, which includes a transferable image encoder and a dual-branch decoder structure.

[0043] Figure 3 This is a schematic diagram of the dual-branch decoder structure designed in this invention. Detailed Implementation

[0044] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0045] With the rapid development of remote sensing sensors and satellites, acquiring high-resolution remote sensing images has become much easier. Semantic segmentation of remote sensing images has been applied in geographic information applications, including urban planning, land resource management, and disaster assessment. Remote sensing images have a significantly different perspective compared to images captured in the natural environment. Natural images are typically captured from a first-person ground view, while remote sensing images are captured from a top-down perspective. Furthermore, remote sensing images usually contain far more objects than natural images, and these objects vary considerably in scale. Therefore, remote sensing semantic segmentation requires stronger feature extraction capabilities and multi-scale feature representation than traditional image segmentation.

[0046] Traditional methods for semantic segmentation of remote sensing images primarily rely on manually designed feature extractors and classifiers, resulting in a lack of generalization ability across diverse remote sensing scenarios. Some classic segmentation models based on convolutional neural networks (CNNs) have achieved significant success in remote sensing image segmentation tasks. However, most of these works use only deep features for prediction, ignoring scale variations. Furthermore, some studies have shown that simple fusion strategies can also lead to redundant feature information.

[0047] In recent years, visual language models have received widespread attention in the field of computer vision. These visual language models can acquire general knowledge and reduce dependence on specific data by pre-training on large-scale image-text datasets. However, due to the domain differences in image distribution, directly applying general visual language models to downstream remote sensing tasks may lead to performance degradation. Some researchers have fine-tuned visual-language models on small-scale remote sensing image-text datasets, demonstrating excellent generalization performance in a range of downstream tasks, including few-sample image classification, image-text retrieval, change detection, and open-vocabulary object detection. However, exploratory work on the application of visual-language models in downstream tasks of remote sensing semantic segmentation is still lacking. Therefore, this invention designs a remote sensing semantic segmentation framework based on an encoder-decoder structure of a visual-language model.

[0048] Combination Figure 1 As shown, the remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder of the present invention includes the following steps:

[0049] Step 1: Collect publicly available remote sensing image data, construct a large-scale remote sensing image-text dataset, and pre-train the image encoder and text encoder of the vision-language model;

[0050] Step 2: Construct an encoder-decoder remote sensing semantic segmentation framework and design a dual-branch weighted encoder structure, which includes a multi-scale feature fusion decoder and a feature difference aggregation decoder.

[0051] Step 3: Transfer the image encoder of the pre-trained visual-language model obtained in Step 1 to the remote sensing semantic segmentation framework in Step 2 as the encoder to extract image features;

[0052] Step 4: Train the entire remote sensing semantic segmentation framework using a large-scale public remote sensing semantic segmentation dataset. The dual-branch decoder is learned independently using two loss functions, each with cross-entropy loss.

[0053] Step 5: Set up dual-branch decoders with different weight constraints to improve the segmentation accuracy of the entire segmentation framework;

[0054] Step 6: Save the weights of the remote sensing semantic segmentation framework trained in Step 5. During the inference phase, load the weight parameters to obtain the visualized segmentation results.

[0055] The specific process of step 1 is as follows:

[0056] 1.1 Collect publicly available remote sensing image data from multiple sources. These images may cover different resolutions, different scenes (such as cities, forests, farmland, etc.), and remote sensing images acquired at different altitudes. This diverse data source helps the model adapt to diverse remote sensing scenes, thereby improving its generalization ability across various scenarios. For each sample image X, a corresponding image text description Y is generated using a multimodal large language model (such as CLIP, BLIP, etc.). caption These text descriptions provide a semantic global overview of each image, describing its content such as terrain features, environmental conditions, and building types. Next, a pre-trained image-text comparison model is used to calculate the sample image X and its corresponding text description Y. caption The similarity score is used to evaluate the consistency between the text description and the image content. Based on the similarity score, high-quality image-text pairs are selected to ensure that the descriptions accurately reflect the key information of the image.

[0057] 1.2 After selecting high-quality image-text pairs, the model is further fine-tuned based on this data. Through fine-tuning of the visual encoder and text encoder using the InfoNCE contrastive loss function, the model will gradually improve its ability to capture remote sensing image features. The formula for calculating the contrastive loss function is as follows:

[0058]

[0059] Where N is the batch size, sim represents the similarity function, and τ is the learnable temperature parameter.

[0060] The specific process of step 2 is as follows:

[0061] 2.1 A remote sensing semantic segmentation framework based on an encoder-decoder architecture is constructed, mainly consisting of an image encoder and a dual-branch weighted decoder structure. The image encoder structure is identical to that in the vision-language model. The dual-branch weighted decoder structure includes a multi-scale feature fusion decoder and a feature difference aggregation decoder. The multi-scale feature fusion decoder integrates features at different scales to reduce the loss of semantic information in multi-scale features. The feature difference aggregation decoder, as an auxiliary branch, captures the differences between features at different levels, thereby enhancing feature complementarity.

[0062] 2.2 Multi-scale Feature Fusion Decoder performs multi-scale fusion through simple convolution and upsampling to obtain multi-scale features. The specific calculation formula is as follows:

[0063]

[0064] Where up represents the upsampling operation and conv represents the convolution function.

[0065] 2.3 The feature difference aggregation decoder captures the differences between multi-scale feature maps by subtracting feature differences, and the resulting difference features are: The specific calculation formula is as follows:

[0066]

[0067] Where | represents absolute value operation, up represents upsampling by 2x through bilinear interpolation, conv represents convolution function, and the convolution kernel k×k={1×1,3×3,5×5}.

[0068] The specific process of step 3 is as follows:

[0069] The image encoder of the pre-trained visual-language model obtained in step 1 is transferred to the remote sensing semantic segmentation framework in step 2 as an encoder to extract image features. This implicitly utilizes the rich remote sensing prior knowledge in the pre-trained visual-language model, providing a good starting point for further training on the downstream remote sensing image semantic segmentation dataset.

[0070] The specific process of step 4 is as follows:

[0071] Cross-entropy loss is used as the loss function to independently learn the dual-branch decoder. The specific calculation formula is as follows:

[0072]

[0073]

[0074] Where L1 and L2 represent the loss functions of the multi-scale fusion decoder and the feature difference aggregation decoder, respectively. i This represents the true label of the i-th pixel. and represents the predictions of the multi-scale fusion decoder and the feature difference aggregation decoder for the i-th pixel, respectively, and N represents the number of pixels in the training sample image.

[0075] The specific process of step 5 is as follows:

[0076] A dual-branch decoder with different weights is configured to balance the impact of each branch decoder on the final prediction result. The total loss of the entire segmentation framework is composed of their weighted sum. The specific calculation formula is as follows:

[0077] L total = aL1 + (1-a)L2

[0078] Where L1 and L2 are the losses of the dual-branch decoder in step 4, and α and 1-α are their respective weights.

[0079] The specific process of step 6 is as follows:

[0080] Save the weights of the remote sensing semantic segmentation framework trained in step 5. In the inference stage, input the test sample image into the segmentation framework to obtain the predicted segmentation result. Finally, visualize the predicted segmentation mask as the final output of this invention.

[0081] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder, characterized in that, Includes the following steps: Step 1: Collect publicly available remote sensing image data, construct a large-scale remote sensing image-text dataset, and pre-train the image encoder and text encoder of the vision-language model; Step 2: Construct an encoder-decoder remote sensing semantic segmentation framework and design a dual-branch weighted encoder structure, which includes a multi-scale feature fusion decoder and a feature difference aggregation decoder. Step 3: Transfer the image encoder of the pre-trained visual-language model obtained in Step 1 to the remote sensing semantic segmentation framework in Step 2 as the encoder to extract image features; Step 4: Train the entire remote sensing semantic segmentation framework using a large-scale public remote sensing semantic segmentation dataset. The dual-branch decoder is learned independently using two loss functions, each with cross-entropy loss. Step 5: Set up dual-branch decoders with different weight constraints to improve the segmentation accuracy of the entire segmentation framework; Step 6: Save the weights of the remote sensing semantic segmentation framework trained in Step 5. During the inference phase, load the weight parameters to obtain the visualized segmentation results.

2. The remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder according to claim 1, characterized in that, The specific process of step 1 is as follows: 1.1 Collect publicly available remote sensing image data from multiple sources. These images may cover different resolutions, different scenes (such as cities, forests, farmland, etc.), and remote sensing images acquired at different altitudes. This diverse data source helps the model adapt to diverse remote sensing scenes, thereby improving its generalization ability across various scenarios. For each sample image X, a corresponding image text description Y is generated using a multimodal large language model (such as CLIP, BLIP, etc.). caption These text descriptions provide a semantic global overview of each image, describing its content such as terrain features, environmental conditions, and building types. Next, a pre-trained image-text comparison model is used to calculate the sample image X and its corresponding text description Y. caption The similarity score is used to evaluate the consistency between the text description and the image content. Based on the similarity score, high-quality image-text pairs are selected to ensure that the descriptions accurately reflect the key information of the image. 1.2 After selecting high-quality image-text pairs, the model is further fine-tuned based on this data. Through fine-tuning of the visual encoder and text encoder using the InfoNCE contrastive loss function, the model will gradually improve its ability to capture remote sensing image features. The formula for calculating the contrastive loss function is as follows: Where N is the batch size, sim represents the similarity function, and τ is the learnable temperature parameter.

3. The remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder according to claim 1, characterized in that, The specific process of step 2 is as follows: 2.1 A remote sensing semantic segmentation framework based on an encoder-decoder architecture is constructed, mainly consisting of an image encoder and a dual-branch weighted decoder structure. The image encoder structure is identical to that in the vision-language model. The dual-branch weighted decoder structure includes a multi-scale feature fusion decoder and a feature difference aggregation decoder. The multi-scale feature fusion decoder integrates features at different scales to reduce the loss of semantic information in multi-scale features. The feature difference aggregation decoder, as an auxiliary branch, captures the differences between features at different levels, thereby enhancing feature complementarity. 2.2 Multi-scale Feature Fusion Decoder performs multi-scale fusion through simple convolution and upsampling to obtain multi-scale features. The specific calculation formula is as follows: Where up represents the upsampling operation and conv represents the convolution function. 2.3 The feature difference aggregation decoder captures the differences between multi-scale feature maps by subtracting feature differences, and the resulting difference features are: The specific calculation formula is as follows: Where || represents absolute value operation, up represents upsampling by 2x through bilinear interpolation, conv represents convolution function, and the convolution kernel k×k={1×1,3×3,5×5}.

4. The remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder according to claim 1, characterized in that, The specific process of step 3 is as follows: The image encoder of the pre-trained visual-language model obtained in step 1 is transferred to the remote sensing semantic segmentation framework in step 2 as an encoder to extract image features. This implicitly utilizes the rich remote sensing prior knowledge in the pre-trained visual-language model, providing a good starting point for further training on the downstream remote sensing image semantic segmentation dataset.

5. The remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder according to claim 1, characterized in that, The specific process of step 4 is as follows: Cross-entropy loss is used as the loss function to independently learn the dual-branch decoder. The specific calculation formula is as follows: Where L1 and L2 represent the loss functions of the multi-scale fusion decoder and the feature difference aggregation decoder, respectively. i This represents the true label of the i-th pixel. and represents the predictions of the multi-scale fusion decoder and the feature difference aggregation decoder for the i-th pixel, respectively, and N represents the number of pixels in the training sample image.

6. The remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder according to claim 1, characterized in that, The specific process of step 5 is as follows: A dual-branch decoder with different weights is configured to balance the impact of each branch decoder on the final prediction result. The total loss of the entire segmentation framework is composed of their weighted sum. The specific calculation formula is as follows: L total =αL1+(1-ɑ)L2 Where L1 and L2 are the losses of the dual-branch decoder in step 4, and α and 1-α are their respective weights.

7. The remote sensing semantic segmentation method based on a visual-language model encoder and a dual-branch weighted decoder according to claim 1, characterized in that, The specific process of step 6 is as follows: Save the weights of the remote sensing semantic segmentation framework trained in step 5. In the inference stage, input the test sample image into the segmentation framework to obtain the predicted segmentation result. Finally, visualize the predicted segmentation mask as the final output of this invention.

Images