A method, system, device, and medium for hierarchical land cover segmentation of large-scene remote sensing images based on multi-source pre-trained model fusion decision-making.

Abstract

A hierarchical land cover segmentation method, system, device, and medium for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making are disclosed. The method includes: constructing a remote sensing image pre-training dataset; training an encoder-decoder structure pre-training model using a mask reconstruction strategy to obtain remote sensing image pre-training weights; constructing a remote sensing image fine-tuning dataset, dividing it into training, validation, and test sets; fine-tuning the encoder-semantic segmentation head structure model using the training and validation sets based on the remote sensing image pre-training weights and natural image pre-training weights, respectively, to obtain two semantic segmentation models; testing the test set using the two semantic segmentation models to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model; fusing the two prediction results based on setted rules to obtain the remote sensing image land cover segmentation result; and implementing the method using a system, device, and medium. This invention has the advantages of high segmentation accuracy and strong robustness.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology and relates to an optical remote sensing image ground feature segmentation method. Specifically, it is a hierarchical ground feature segmentation method, system, equipment and medium for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, which can be used in infrastructure monitoring, natural resource management, environmental monitoring and other fields.

[0002] Background Area

[0003] Land cover segmentation in optical remote sensing imagery has profound and multifaceted significance. It can clearly separate different land cover categories from complex images, providing detailed and accurate data support for land resource management, urban development research, disaster monitoring, and emergency response. In recent years, pre-training deep learning models has been able to uncover the latent features of massive amounts of data and enhance the model's ability to fit the data. However, most current pre-training is for natural images, which is difficult to apply effectively to remote sensing scenarios.

[0004] Natural images often focus on everyday scenes, people, and objects, while optical remote sensing imagery focuses more on information related to geography, environment, and land use. The two differ significantly in their emphasis and feature representation, thus directly using natural images to pre-train models in remote sensing imagery visual tasks still faces performance bottlenecks. Furthermore, existing remote sensing images are difficult to acquire, have high labeling costs, and exhibit certain unique characteristics in their scenes, leading to limited model generalization performance when directly pre-trained on them. Currently, more and more researchers in related fields are focusing on how to utilize multi-source data to provide complementary information for models, thereby improving model accuracy and reliability and enhancing generalization ability. Simultaneously, introducing other relevant data sources can also compensate for the insufficient amount of data from a single source.

[0005] Patent application CN118314353A discloses a remote sensing image segmentation method based on bi-branch multi-scale feature fusion. This method addresses the shortcomings of existing remote sensing image segmentation methods, such as their inability to effectively handle ground features at different scales, achieve accurate segmentation without consuming excessive computation and memory, and suffer from unstable segmentation results. However, while this method expands the scale of the remote sensing image source, its feature extraction based solely on remote sensing images results in poor generalization ability and fault tolerance when facing complex scenes. Furthermore, this method requires high data consistency and synchronization, necessitating the acquisition of remote sensing images of the same area at different scales. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device, and medium for hierarchical land cover segmentation of large-scene remote sensing images based on multi-source pre-trained model fusion decision-making. The method combines the general capabilities of natural image pre-trained models with the specific capabilities of remote sensing image pre-trained models through fusion decision-making, thereby solving the problems of low accuracy of land cover segmentation categories in large-scene remote sensing images and high requirements for data consistency and synchronization in the prior art.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making includes the following steps:

[0009] Step 1: Construct the remote sensing image pre-training dataset D pre : Preprocessing and cropping massive amounts of optical remote sensing images to obtain a remote sensing image pre-training dataset D pre ;

[0010] Step 2: Constructing a fine-tuned remote sensing image dataset: First, construct a set of typical land cover categories in a large scene and merge the categories of labeled optical remote sensing images; second, perform non-overlapping cropping on the merged optical remote sensing images and their labels to obtain a fine-tuned remote sensing image dataset, and divide it into training set, validation set and test set according to proportions.

[0011] Step 3: Model pre-training: Using the remote sensing image pre-training dataset D obtained in Step 1 pre The pre-trained model of the encoder-decoder structure is trained using a mask reconstruction strategy to obtain the pre-trained weights of the remote sensing image.

[0012] Step 4: Model Fine-tuning and Inference: First, based on the pre-trained weights of remote sensing images obtained in Step 3 and the pre-trained weights of natural images obtained from the ImageNet dataset, the encoder-semantic segmentation head model is fine-tuned using the training and validation sets generated in Step 2 to obtain two semantic segmentation models with the same structure but different weights. Second, the two semantic segmentation models are tested on the test set generated in Step 2 to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model.

[0013] Step 5: Rule-based fusion decision: Based on the set rules, the prediction results of the natural image pre-trained model and the remote sensing image pre-trained model in Step 4 are fused to obtain the remote sensing image land cover segmentation result.

[0014] Step 6: Performance Evaluation: Calculate the evaluation index of each category and the overall quality of the remote sensing image land cover segmentation results obtained in Step 5, and quantify the fusion performance.

[0015] Step 1 involves constructing the remote sensing image pre-training dataset D. pre The specific process is as follows:

[0016] Step 1.1: Perform preprocessing operations such as data screening and cleaning on massive optical remote sensing images to obtain high-quality remote sensing image data;

[0017] Step 1.2: The high-quality remote sensing image data obtained in Step 1.1 is cropped non-overlapping to a size of 512*512 to obtain the remote sensing image pre-training dataset D. pre .

[0018] The specific process of constructing the fine-tuning dataset in step 2 is as follows:

[0019] Step 2.1: Construct a hierarchical category system for typical land features in a large scene. The first-level categories include, but are not limited to: planted land, forest and grassland cover, buildings, transportation, structures, artificial excavated land, bare ground, and water area; the second-level categories include: terraced fields, paddy fields, dry land, orchards, tea gardens, wetlands, rural villages, educational land, railways, highways, other roads, other transportation facilities, silt dams, city walls (ancient towns), greenhouses, chemical plants, solar panels, open-pit metal mines, river sand dredging plants, gullies, other bare ground, canals, lakes, trees, shrubs, artificial grassland, natural grassland, high-rise office buildings, factory areas, residential areas, regulating reservoirs, aquaculture water surfaces, and planted water surfaces.

[0020] Step 2.2: Select labeled optical remote sensing images, merge categories according to the category system constructed in Step 2.1, and perform non-overlapping cropping according to a size of 512*512 to obtain the remote sensing image fine-tuning dataset D. seg The dataset was divided into training, validation, and test sets in a ratio of 8:1:1.

[0021] The specific process of model pre-training in step 3 is as follows:

[0022] The pre-trained model of the encoder-decoder structure includes encoder E and decoder D;

[0023] Step 3.1: Model Input: The remote sensing image pre-training dataset constructed in Step 1.2 Sample D after masking pre As input to the pre-trained model of the encoder-decoder structure:

[0024]

[0025] Where Ψ(·) represents the pre-trained model, and θ represents the optimization parameters in the model;

[0026] Step 3.2: Define encoder E: The backbone network used by encoder E is Swin-S. The Swin-S network consists of 4 stages, and the number of Swin Transformer blocks contained in each stage from beginning to end are 2, 2, 18, and 2 respectively.

[0027] F pre =Encoder(D pre )

[0028] Where Encoder(·) represents the encoder network, F pre This indicates the characteristics of the encoder output;

[0029] Step 3.3: Define decoder D: Decoder D uses a single fully connected layer, and its output is the same as that of sample D. pre Reconstruction results with the same dimensions:

[0030]

[0031] Where Decoder(·) represents the decoder network, Indicates the reconstruction result;

[0032] Step 3.4: Design a mask reconstruction strategy: Select the mask block size and mask rate, and use the selected mask block size to randomly mask a portion of the input data according to the mask rate;

[0033] Step 3.5: Construct the reconstruction loss, using the mean absolute error as the loss function to calculate the difference between the original image and the reconstructed image:

[0034]

[0035] Where L1(·) represents the mean absolute error, and y is the true value. This is the predicted value, and n is the number of samples;

[0036] Step 3.6: Pre-train the pre-trained model and save the encoder weights after training as pre-training weights for remote sensing images.

[0037] The specific process of model fine-tuning and inference in step 4 is as follows:

[0038] The encoder-semantic segmentation head structure model includes encoder E and semantic segmentation head H;

[0039] Step 4.1: Define the semantic segmentation head H: Use 1×1 convolution to reduce the channel dimension of the shallow features of the input, i.e., the output of the first stage of encoder E; use Atrous Spatial Pyramid Pooling (ASPP) to extract the deep features of the input, i.e., the output of the fourth stage of encoder E, at multiple scales; finally, concatenate the processed shallow features and deep features and input them into the classifier to obtain the final output.

[0040] Step 4.2: Connect the encoder E from step 3.2 with the semantic segmentation head H, and use the output of stage 1 of encoder E as the shallow feature input and the output of stage 4 as the deep feature input;

[0041] Step 4.3: Use the pre-trained weights of remote sensing images obtained in Step 3 and the pre-trained weights of natural images obtained from the ImageNet dataset as the initial weights of encoder E, and fine-tune them using the fine-tuning dataset of remote sensing images in Step 2.2 to obtain two semantic segmentation models with the same structure but different weights.

[0042] Step 4.4: Take samples D from the test set generated in Step 2.2 respectively. seg The two semantic segmentation models fine-tuned in step 4.3 are used to predict the results, resulting in the predicted category probability distribution Pro. out And the prediction result Pred out :

[0043]

[0044] in, represents the semantic segmentation head network, softmax(·) represents the normalized exponential function, and argmax(·) represents the maximum value function.

[0045] The specific process of rule-based fusion decision-making in step 5 is as follows:

[0046] Step 5.1: Pixel-level maximum probability calculation: Calculate the class probability distribution Pro obtained in Step 4.4. out The maximum probability of each pixel is calculated to obtain the maximum probability of each pixel:

[0047]

[0048] Among them, Pro max This represents the maximum probability of the pixel, and max(·) represents the function for calculating the maximum value.

[0049] Step 5.2: Construct pixel contrast rules and class injection: Based on the maximum probability of a pixel obtained in Step 5.1, perform pixel-level probability comparison. If the probability of the natural image at that pixel is greater than the maximum probability of the remote sensing image under the set rules, then the prediction of the natural image pre-trained model is injected into that pixel as the result; otherwise, the prediction of the remote sensing image pre-trained model is injected into that pixel as the result.

[0050]

[0051] Among them, Pred fusion Pred represents the fused prediction result. pre Pred represents the prediction result of the pre-trained model of remote sensing imagery. ImageNet This represents the prediction results of a pre-trained model for natural images. This represents the maximum probability of the remote sensing image pre-trained model at this pixel. ρ represents the maximum probability of the pre-trained model for a natural image at that pixel, and ρ represents the rule threshold.

[0052] The evaluation indicators for each category in step 6 include:

[0053] Precision:

[0054]

[0055] Where TP represents True Positive (the number of samples correctly predicted as positive) and FP represents False Positive (the number of samples incorrectly predicted as positive);

[0056] Intersection over Union (IoU):

[0057]

[0058] Where TP represents True Positive (the number of samples correctly predicted as positive), FP represents False Positive (the number of samples incorrectly predicted as positive), and FN represents False Negative (the number of samples incorrectly predicted as negative).

[0059] Recall:

[0060]

[0061] Where TP represents True Positive (the number of samples correctly predicted as positive) and FN represents False Negative (the number of samples incorrectly predicted as negative);

[0062] F1 score:

[0063]

[0064] The overall quality evaluation indicators include:

[0065] Mean Crossover Union Ratio (MIoU):

[0066]

[0067] Where k represents the number of categories, p ij This represents the number of pixels that originally belonged to class i but were predicted to belong to class j.

[0068] This invention also provides a hierarchical land cover segmentation system for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, comprising:

[0069] The remote sensing image pre-training dataset construction module is used to preprocess and crop massive amounts of optical remote sensing images to obtain the remote sensing image pre-training dataset D. pre ;

[0070] The remote sensing image fine-tuning dataset construction module is used to construct a set of typical land cover categories in a large scene, merge the categories of labeled optical remote sensing images, and perform non-overlapping cropping on the merged optical remote sensing images and their labels to obtain the remote sensing image fine-tuning dataset, and divide it into training set, validation set and test set according to the proportion.

[0071] The model pre-training module is used to pre-train the remote sensing imagery dataset D. pre The pre-trained model of the encoder-decoder structure is trained using a mask reconstruction strategy to obtain the pre-trained weights of the remote sensing image.

[0072] The model fine-tuning and inference module is used to fine-tune the encoder-semantic segmentation head structure model using training and validation sets, based on pre-trained weights from remote sensing images and pre-trained weights from natural images obtained from the ImageNet dataset. This results in two semantic segmentation models with the same structure but different weights. The two semantic segmentation models are then tested on the test set to obtain the prediction results of the pre-trained natural image model and the pre-trained remote sensing image model.

[0073] The remote sensing image ground feature segmentation module is used to fuse the prediction results of the natural image pre-trained model and the remote sensing image pre-trained model based on the set rules to obtain the remote sensing image ground feature segmentation result.

[0074] The performance evaluation module is used to calculate the evaluation indicators of the quality of each category and the overall quality in the remote sensing image land cover segmentation results, and to quantify the fusion performance.

[0075] This invention also provides a hierarchical land cover segmentation device for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, comprising:

[0076] Memory: A computer program that stores the above-mentioned hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, and is a computer-readable device;

[0077] Processor: Used to implement the hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making when executing the computer program.

[0078] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned method for hierarchical land cover segmentation of large-scene remote sensing images based on multi-source pre-trained model fusion decision-making.

[0079] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0080] 1. This invention uses a multi-source data pre-trained model for modeling, that is, it performs feature extraction and image reconstruction on natural images and remote sensing images respectively. This can combine the general capabilities of the natural image pre-trained model with the specific capabilities of the remote sensing image pre-trained model, thereby enhancing the model's generalization ability in different remote sensing scenarios.

[0081] 2. This invention compares the predicted class probabilities of the natural image pre-trained model and the remote sensing image pre-trained model at the same pixel through rule-based fusion decision. When the class predicted by the natural image pre-trained model is different from the class predicted by the remote sensing image pre-trained model, the more reliable class is adopted as the prediction result, which can effectively remove noise and improve the anomalies in the segmentation results of large-scene remote sensing ground objects in the existing technology.

[0082] In summary, this invention, through multi-source data pre-trained modeling and rule-based fusion decision-making, has the advantages of high segmentation accuracy and strong robustness. Attached Figure Description

[0083] Figure 1 This is a flowchart of the method of the present invention.

[0084] Figure 2 This is an experimental prediction result diagram comparing the present invention with a natural image pre-trained model and a remote sensing image pre-trained model on a test set with an image size of 512×512.

[0085] Figure 3 This is an experimental prediction result diagram comparing the high-resolution remote sensing images and natural image pre-training models with the remote sensing image pre-training model in complex scenes of Guanzhong and southern Shaanxi. Detailed Implementation

[0086] The present invention will now be further described with reference to the accompanying drawings.

[0087] The purpose of this invention is to overcome the shortcomings of existing technologies. In view of the poor scene adaptability of natural image pre-trained models and their inability to accurately identify complex ground features, as well as the insufficient generalization ability of remote sensing image pre-trained models, this invention proposes a hierarchical ground feature segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, in order to solve the problem of low accuracy of existing technologies in large scenes.

[0088] like Figure 1 As shown, a hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making includes the following steps:

[0089] Step 1: Construct the remote sensing image pre-training dataset D pre : Preprocessing and cropping massive amounts of optical remote sensing images to obtain a remote sensing image pre-training dataset D pre ;

[0090] Step 1 involves constructing the remote sensing image pre-training dataset D. pre The specific process is as follows:

[0091] Step 1.1: Perform preprocessing operations such as data screening and cleaning on a large number of optical remote sensing images of Shaanxi Province to obtain high-quality remote sensing image data;

[0092] Step 1.2: The high-quality remote sensing image data obtained in Step 1.1 is cropped non-overlapping to a size of 515*512 to obtain a pre-training dataset D containing a certain number of remote sensing images. pre .

[0093] Step 2: Constructing a fine-tuned remote sensing image dataset: First, construct a set of typical land cover categories in a large scene and merge the categories of labeled optical remote sensing images; second, perform non-overlapping cropping on the merged optical remote sensing images and their labels to obtain a fine-tuned remote sensing image dataset, and divide it into training set, validation set and test set according to a certain ratio.

[0094] The specific process of constructing the fine-tuning dataset in step 2 is as follows:

[0095] Step 2.1: Construct a hierarchical category system for typical land features in a large scene. The first-level categories include, but are not limited to: planted land, forest and grassland cover, buildings, transportation, structures, artificial excavated land, bare ground, and water area; the second-level categories include: terraced fields, paddy fields, dry land, orchards, tea gardens, wetlands, rural villages, educational land, railways, highways, other roads, other transportation facilities, silt dams, city walls (ancient towns), greenhouses, chemical plants, solar panels, open-pit metal mines, river sand dredging plants, gullies, other bare ground, canals, lakes, trees, shrubs, artificial grassland, natural grassland, high-rise office buildings, factory areas, residential areas, regulating reservoirs, aquaculture water surfaces, and planted water surfaces.

[0096] Step 2.2: Select labeled optical remote sensing images of Shaanxi Province, merge the categories according to the classification system constructed in Step 2.1, and perform non-overlapping cropping according to a size of 512*512 to obtain a fine-tuned dataset D containing a certain number of remote sensing images. seg The dataset was divided into training, validation, and test sets in a ratio of 8:1:1.

[0097] Step 3: Model pre-training: Using the remote sensing image pre-training dataset D obtained in Step 1 pre The pre-trained model of the encoder-decoder structure is trained using a mask reconstruction strategy to obtain the pre-trained weights of the remote sensing image.

[0098] The specific process of model pre-training in step 3 is as follows:

[0099] The pre-trained model of the encoder-decoder structure includes encoder E and decoder D;

[0100] Step 3.1: Model Input: Masked samples D from the remote sensing image pre-training dataset constructed in Step 1.2. pre As input to the pre-trained model of the encoder-decoder structure:

[0101]

[0102] Where Ψ(·) represents the pre-trained model, and θ represents the optimization parameters in the model.

[0103] Step 3.2: Define encoder E: The backbone network used by encoder E in this invention is Swin-S. The Swin-S network mainly consists of 4 stages, and the number of Swin Transformer blocks contained in each stage from front to back are 2, 2, 18, and 2 respectively.

[0104] F pre =Encoder(D pre )

[0105] Where Encoder(·) represents the encoder network, F pre This indicates the characteristics of the encoder output;

[0106] Step 3.3: Define decoder D: Decoder D uses a single fully connected layer, and its output is connected to D. pre Reconstruction results with the same dimensions:

[0107]

[0108] Where Decoder(·) represents the decoder network, Indicates the reconstruction result;

[0109] Step 3.4: Design a mask reconstruction strategy: Select the mask block size and mask rate, and use the selected mask block size to randomly mask a portion of the input data according to the mask rate;

[0110] Step 3.5: Constructing the reconstruction loss. This invention uses the mean absolute error as the loss function to calculate the difference between the original image and the reconstructed image:

[0111]

[0112] Where L1(·) represents the mean absolute error, and y is the true value. is the predicted value, and n is the number of samples.

[0113] Step 3.6: Pre-train the pre-trained model and save the encoder weights after training as pre-training weights for remote sensing images.

[0114] Step 4: Model Fine-tuning and Inference: First, based on the pre-trained weights of remote sensing images obtained in Step 3 and the pre-trained weights of natural images obtained from the ImageNet dataset, the encoder-semantic segmentation head model is fine-tuned using the training and validation sets generated in Step 2 to obtain two semantic segmentation models with the same structure but different weights. Second, the two semantic segmentation models are tested on the test set generated in Step 2 to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model.

[0115] The specific process of model fine-tuning and inference in step 4 is as follows:

[0116] The encoder-semantic segmentation head structure model includes encoder E and semantic segmentation head H;

[0117] Step 4.1: Define the semantic segmentation head H: Use 1×1 convolution to reduce the channel dimension of the input shallow features (output of encoder E in stage 1); use Atrous Spatial Pyramid Pooling (ASPP) to extract the input deep features (output of encoder E in stage 4) at multiple scales; finally, concatenate the processed shallow features and deep features and input them into the classifier to obtain the final output.

[0118] Step 4.2: Connect the encoder E from step 3.2 with the semantic segmentation head H, and use the output of stage 1 of encoder E as the shallow feature input and the output of stage 4 as the deep feature input;

[0119] Step 4.3: Use the pre-trained weights of remote sensing images obtained in Step 3 and the pre-trained weights of natural images obtained from the ImageNet dataset as the initial weights of encoder E, and fine-tune them using the fine-tuning dataset of remote sensing images in Step 2.2 to obtain two semantic segmentation models with the same structure but different weights.

[0120] Step 4.4: Extract samples D from the remote sensing image test set generated in Step 2.2. seg The two semantic segmentation models fine-tuned in step 4.3 are used to predict the results, resulting in the predicted category probability distribution Pro. out And the prediction result Pred out :

[0121]

[0122] in, represents the semantic segmentation head network, softmax(·) represents the normalized exponential function, and argmax(·) represents the maximum value function.

[0123] Step 5: Rule-based fusion decision: Based on the set rules, the prediction results of the natural image pre-trained model and the remote sensing image pre-trained model in Step 4 are fused to obtain higher quality remote sensing image semantic segmentation results.

[0124] The rule-based fusion decision-making approach in step 5 is as follows:

[0125] Step 5.1: Pixel-level maximum probability calculation: Calculate the class probability distribution Pro obtained in Step 4.4. out The maximum probability of each pixel is calculated to obtain the maximum probability of each pixel:

[0126]

[0127] Among them, Pro maxThis represents the maximum probability of the pixel, and max(·) represents the function for calculating the maximum value.

[0128] Step 5.2: Construct pixel contrast rules and class injection: Based on the maximum probability of a pixel obtained in Step 5.1, perform pixel-level probability comparison. If the probability of the natural image at that pixel is greater than the maximum probability of the remote sensing image under the set rules, then the prediction of the natural image pre-trained model is injected into that pixel as the result; otherwise, the prediction of the remote sensing image pre-trained model is injected into that pixel as the result.

[0129]

[0130] Among them, Pred fusion Pred represents the fused prediction result. pre Pred represents the prediction result of the pre-trained model of remote sensing imagery. ImageNet This represents the prediction results of a pre-trained model for natural images. This represents the maximum probability of the remote sensing image pre-trained model at this pixel. ρ represents the maximum probability of the pre-trained model for the natural image at that pixel, and ρ represents the rule threshold. In this invention, ρ = 0.9.

[0131] Step 6: Performance Evaluation: Calculate the evaluation index of each category and the overall quality of the remote sensing image land cover segmentation results obtained in Step 5, and quantify the fusion performance.

[0132] The evaluation indicators for each category in step 6 include:

[0133] Precision:

[0134]

[0135] Where TP represents True Positive (the number of samples correctly predicted as positive) and FP represents False Positive (the number of samples incorrectly predicted as positive);

[0136] Intersection over Union (IoU):

[0137]

[0138] Where TP represents True Positive (the number of samples correctly predicted as positive), FP represents False Positive (the number of samples incorrectly predicted as positive), and FN represents False Negative (the number of samples incorrectly predicted as negative).

[0139] Recall:

[0140]

[0141] Where TP represents True Positive (the number of samples correctly predicted as positive) and FN represents False Negative (the number of samples incorrectly predicted as negative);

[0142] F1 score:

[0143]

[0144] The indicators for overall quality evaluation include:

[0145] Mean Crossover Union Ratio (MIoU):

[0146]

[0147] Where k represents the number of categories, p ij This represents the number of pixels that originally belonged to class i but were predicted to belong to class j.

[0148] The technical problem to be solved by this invention is that existing technologies usually use single-source pre-trained models for single modeling, which makes it impossible to directly apply natural image pre-training to remote sensing image land cover segmentation tasks; at the same time, because remote sensing image pre-training is difficult to mine feature diversity in massive data, it will produce obvious performance bottlenecks when pre-training remote sensing images.

[0149] To address the aforementioned issues, this invention proposes a hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making. Through a novel rule-based fusion decision-making method, this method significantly improves the handling of anomalies in large-scene remote sensing image land cover segmentation results using existing technologies and enhances the robustness of the model.

[0150] like Figure 2 The diagram shows the experimental prediction results of this invention compared with pre-trained models of natural images and remote sensing images on a test set in typical scenarios. The comparison reveals that this invention has significant advantages in classification results for other typical scenarios such as highways, artificial grasslands, orchards, educational land, and factory areas. Figure 2 As can be seen in Scenario 1, the pre-trained model using natural images performs poorly in distinguishing between educational land and factory areas, while the pre-trained model using remote sensing images fails to extract the artificial grassland category. This invention, through fusion decision-making, allows the two models to compensate for their respective shortcomings, resulting in a more accurate classification result. Figure 2 As can be seen in Scenario 2, both the natural image pre-trained model and the remote sensing image pre-trained model have problems with noise in local areas and inconsistent classification results. However, the present invention effectively removes noise and has better noise suppression ability and robustness.

[0151] like Figure 3 The image shows the generalization prediction results of this invention compared with natural image pre-training models and remote sensing image pre-training models on high-resolution remote sensing image data of complex scenes in Guanzhong and southern Shaanxi. Scene 1 is a high-resolution remote sensing image of a complex scene in Guanzhong. The comparison shows that this invention removes noise distributed in urban and forest areas in the classification results of natural image pre-training models and remote sensing image pre-training models, while accurately classifying most categories in urban areas. Scene 2 is a high-resolution remote sensing image of a complex scene in southern Shaanxi. The comparison shows that this invention provides clearer boundaries between trees and dry land, while removing some areas that were incorrectly identified as water bodies and background.

[0152] As shown in Table 1, the quantitative results of this invention demonstrate that, compared with the pre-trained models for natural images and remote sensing images, the overall mIoU (%) is improved by 1.43 and 4.09, respectively. Furthermore, this invention improves Precision, Recall, and IoU (Intersection over Union) in most categories, particularly for categories such as "Regulating Reservoir," "Aquaculture Water Surface," "Railway," "Other Highways," "Other Transportation Facilities," "Chemical Plant," and "Factory Area." The IoU (%) improvements relative to the pre-trained models for natural images are 3.38, 2.21, 3.37, 2.72, 2.42, 3.91, and 2.02, respectively, and relative to the pre-trained models for remote sensing images are 3.16, 2.94, 5.94, 1.82, 2.18, 3.13, and 3.18, respectively. Therefore, this invention effectively improves the accuracy of feature segmentation in large-scene remote sensing images.

[0153] Table 1 compares the evaluation metrics of this invention (bottom), the natural image pre-trained model (top), and the remote sensing image pre-trained model (middle) on the same dataset.

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160] This invention also provides a hierarchical land cover segmentation system for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, comprising:

[0161] The remote sensing image pre-training dataset construction module is used to perform preprocessing and cropping of massive optical remote sensing images in step 1 to obtain the remote sensing image pre-training dataset D. pre ;

[0162] The remote sensing image fine-tuning dataset construction module is used to construct a set of typical land cover categories in a large scene in step 2, merge the categories of labeled optical remote sensing images, and perform non-overlapping cropping on the merged optical remote sensing images and their labels to obtain the remote sensing image fine-tuning dataset, and divide it into training set, validation set and test set according to proportions.

[0163] The model pre-training module is used in step 3 to implement the pre-training dataset D of remote sensing images obtained in step 1. pre The pre-trained model of the encoder-decoder structure is trained using a mask reconstruction strategy to obtain the pre-trained weights of the remote sensing image.

[0164] The model fine-tuning and inference module is used to implement the fine-tuning of the encoder-semantic segmentation head structure model in step 4 based on the remote sensing image pre-training weights obtained in step 3 and the natural image pre-training weights obtained from the ImageNet dataset, using the training set and validation set generated in step 2, to obtain two semantic segmentation models with the same structure but different weights; the two semantic segmentation models are used to test the test set generated in step 2 to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model;

[0165] The remote sensing image ground feature segmentation module is used to perform fusion decision on the prediction results of the natural image pre-training model and the remote sensing image pre-training model in step 4 based on the set rules in step 5, so as to obtain the remote sensing image ground feature segmentation result.

[0166] The performance evaluation module is used to calculate the evaluation indicators of each category and the overall quality of the remote sensing image land cover segmentation results obtained in step 5 in step 6, and to quantify the fusion performance.

[0167] This invention also provides a hierarchical land cover segmentation device for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, comprising:

[0168] Memory: A computer program that stores the above-mentioned hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, and is a computer-readable device;

[0169] Processor: Used to implement the hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making when executing the computer program.

[0170] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned method for hierarchical land cover segmentation of large-scene remote sensing images based on multi-source pre-trained model fusion decision-making.

Claims

1. A hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, characterized in that, Includes the following steps: Step 1: Construct a remote sensing image pre-training dataset Preprocessing and cropping of massive amounts of optical remote sensing images yields a remote sensing image pre-training dataset. Step 1 involves constructing a remote sensing image pre-training dataset. The specific process is as follows: Step 1.1: Perform preprocessing operations such as data screening and cleaning on massive optical remote sensing images to obtain high-quality remote sensing image data; Step 1.2: The high-quality remote sensing image data obtained in Step 1.1 is cropped non-overlapping to a size of 512*512 to obtain the remote sensing image pre-training dataset. Step 2: Constructing a fine-tuned remote sensing image dataset: First, construct a set of typical land cover categories in a large scene and merge the categories of labeled optical remote sensing images; second, perform non-overlapping cropping on the merged optical remote sensing images and their labels to obtain a fine-tuned remote sensing image dataset, and divide it into training set, validation set and test set according to proportions. The specific process of constructing the fine-tuning dataset in step 2 is as follows: Step 2.1: Construct a hierarchical classification system for typical features in a large-scale scene; Step 2.2: Select labeled optical remote sensing images, merge categories according to the category system constructed in Step 2.1, and perform non-overlapping cropping according to a size of 512*512 to obtain the remote sensing image fine-tuning dataset. It is divided into training set, validation set and test set in a ratio of 8:1:1; Step 3: Model pre-training: Use the remote sensing image pre-training dataset obtained in Step 1. The pre-trained model of the encoder-decoder structure is trained using a mask reconstruction strategy to obtain the pre-trained weights of the remote sensing image. The specific process of model pre-training in step 3 is as follows: The pre-trained model of the encoder-decoder structure includes encoder E and decoder D; Step 3.1: Model Input: The remote sensing image pre-training dataset constructed in Step 1.2 The masked sample D pre As input to the pre-trained model of the encoder-decoder structure: Where Ψ(·) represents the pre-trained model, and θ represents the optimization parameters in the model; Step 3.2: Define encoder E: The backbone network used by encoder E is Swin-S. The Swin-S network consists of 4 stages, and the number of Swin Transformer blocks contained in each stage from beginning to end are 2, 2, 18, and 2 respectively. F pre =Encoder(D pre ) Where Encoder(·) represents the encoder network, F pre This indicates the characteristics of the encoder output; Step 3.3: Define decoder D: Decoder D uses a single fully connected layer, and its output is the same as that of sample D. pre Reconstruction results with the same dimensions: Where Decoder(·) represents the decoder network, Indicates the reconstruction result; Step 3.4: Design a mask reconstruction strategy: Select the mask block size and mask rate, and use the selected mask block size to randomly mask a portion of the input data according to the mask rate; Step 3.5: Construct the reconstruction loss, using the mean absolute error as the loss function to calculate the difference between the original image and the reconstructed image: Where L1(·) represents the mean absolute error, and y is the true value. This is the predicted value, and n is the number of samples; Step 3.6: Pre-train the pre-trained model and save the encoder weights after training as pre-training weights for remote sensing images; Step 4: Model Fine-tuning and Inference: First, based on the pre-trained weights of remote sensing images obtained in Step 3 and the pre-trained weights of natural images obtained from the ImageNet dataset, the encoder-semantic segmentation head model is fine-tuned using the training and validation sets generated in Step 2 to obtain two semantic segmentation models with the same structure but different weights. Second, the two semantic segmentation models are tested on the test set generated in Step 2 to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model. The specific process of model fine-tuning and inference in step 4 is as follows: The encoder-semantic segmentation head structure model includes encoder E and semantic segmentation head H; Step 4.1: Define the semantic segmentation head H: Use 1×1 convolution to reduce the channel dimension of the shallow features of the input, i.e., the output of the first stage of encoder E; use Atrous Spatial Pyramid Pooling (ASPP) to extract the deep features of the input, i.e., the output of the fourth stage of encoder E, at multiple scales; finally, concatenate the processed shallow features and deep features and input them into the classifier to obtain the final output. Step 4.2: Connect the encoder E from step 3.2 with the semantic segmentation head H, and use the output of stage 1 of encoder E as the shallow feature input and the output of stage 4 as the deep feature input; Step 4.3: Use the pre-trained weights of remote sensing images obtained in Step 3 and the pre-trained weights of natural images obtained from the ImageNet dataset as the initial weights of encoder E, and fine-tune them using the fine-tuning dataset of remote sensing images in Step 2.2 to obtain two semantic segmentation models with the same structure but different weights. Step 4.4: Process the samples from the test set generated in Step 2.2... The two semantic segmentation models fine-tuned in step 4.3 are used to predict the results, resulting in the predicted category probability distribution Pro. out And the prediction result Pred out : in, This represents the semantic segmentation head network, softmax(·) represents the normalization exponential function, and argmax(·) represents the maximum value function of the independent variable. Step 5: Rule-based fusion decision: Based on the set rules, the prediction results of the natural image pre-trained model and the remote sensing image pre-trained model in Step 4 are fused to obtain the remote sensing image land cover segmentation result. Step 6: Performance Evaluation: Calculate the evaluation index of each category and the overall quality of the remote sensing image land cover segmentation results obtained in Step 5, and quantify the fusion performance.

2. The hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making as described in claim 1, characterized in that, The specific process of rule-based fusion decision-making in step 5 is as follows: Step 5.1: Pixel-level maximum probability calculation: Calculate the class probability distribution Pro obtained in Step 4.

4. out The maximum probability of each pixel is calculated to obtain the maximum probability of each pixel: Among them, Pro max This represents the maximum probability of the pixel, and max(·) represents the function for calculating the maximum value. Step 5.2: Construct pixel contrast rules and class injection: Based on the maximum probability of a pixel obtained in Step 5.1, perform pixel-level probability comparison. If the probability of the natural image at that pixel is greater than the maximum probability of the remote sensing image under the set rules, then the prediction of the natural image pre-trained model is injected into that pixel as the result; otherwise, the prediction of the remote sensing image pre-trained model is injected into that pixel as the result. Among them, Pred fusion Pred represents the fused prediction result. pre Pred represents the prediction result of the pre-trained model of remote sensing imagery. ImageNet This represents the prediction results of a pre-trained model for natural images. This represents the maximum probability of the remote sensing image pre-trained model at this pixel. ρ represents the maximum probability of the pre-trained model for a natural image at that pixel, and ρ represents the rule threshold.

3. The hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making as described in claim 1, characterized in that, The evaluation indicators for each category in step 6 include: Precision: Where TP represents a true positive and FP represents a false positive; Intersection over Union (IoU): Where TP represents true positives, FP represents false positives, and FN represents false negatives; Recall: Where TP represents a true negative and FN represents a false negative; F1 score: The overall quality evaluation indicators include: Mean Crossover Union Ratio (MIoU): Where k represents the number of categories, p ij This represents the number of pixels that originally belonged to class i but were predicted to belong to class j.

4. A hierarchical land cover segmentation system for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, based on the method described in any one of claims 1 to 3, characterized in that, include: The remote sensing image pre-training dataset construction module is used to preprocess and crop massive amounts of optical remote sensing images to obtain remote sensing image pre-training datasets. The remote sensing image fine-tuning dataset construction module is used to construct a set of typical land cover categories in a large scene, merge the categories of labeled optical remote sensing images, and perform non-overlapping cropping on the merged optical remote sensing images and their labels to obtain the remote sensing image fine-tuning dataset, and divide it into training set, validation set and test set according to the proportion. The model pre-training module is used to pre-train a dataset of remote sensing images. The pre-trained model of the encoder-decoder structure is trained using a mask reconstruction strategy to obtain the pre-trained weights of the remote sensing image. The model fine-tuning and inference module is used to fine-tune the encoder-semantic segmentation head structure model using training and validation sets, based on pre-trained weights from remote sensing images and pre-trained weights from natural images obtained from the ImageNet dataset. This results in two semantic segmentation models with the same structure but different weights. The two semantic segmentation models are then tested on the test set to obtain the prediction results of the pre-trained natural image model and the pre-trained remote sensing image model. The remote sensing image ground feature segmentation module is used to fuse the prediction results of the natural image pre-trained model and the remote sensing image pre-trained model based on the set rules to obtain the remote sensing image ground feature segmentation result. The performance evaluation module is used to calculate the evaluation indicators of the quality of each category and the overall quality in the remote sensing image land cover segmentation results, and to quantify the fusion performance.

5. A hierarchical land cover segmentation device for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, characterized in that, include: Memory: A computer program for a hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making, as described in any one of claims 1-3, is a computer-readable device; Processor: Used to implement the hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making as described in any one of claims 1-3 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, can implement the hierarchical land cover segmentation method for large-scene remote sensing images based on multi-source pre-trained model fusion decision-making as described in any one of claims 1-3.

Images