Multi-source pre-training model fusion decision-based large-scene remote sensing image hierarchical ground feature segmentation method, system and device, and medium

Abstract

The invention discloses a multi-source pre-training model fusion decision-based large-scene remote sensing image hierarchical ground feature segmentation method, system and device, and a medium. The method comprises the following steps: constructing a remote sensing image pre-training data set; training the pre-training model of the encoder-decoder structure by adopting a mask reconstruction strategy to obtain a remote sensing image pre-training weight; constructing a remote sensing image fine tuning data set, and dividing training, verification and test sets; finely adjusting the model of the encoder-semantic segmentation head structure by using a training set and a verification set on the basis of the remote sensing image pre-training weight and the natural image pre-training weight to obtain two semantic segmentation models; respectively testing the test set by using two semantic segmentation models to obtain prediction results of the natural image pre-training model and the remote sensing image pre-training model; based on a set rule, performing fusion decision on the two prediction results to obtain a remote sensing image ground object segmentation result; the system, the device and the medium implement the method. The method has the advantages of high segmentation precision and strong robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and relates to a method for segmenting objects in optical remote sensing images, specifically a method, system, equipment and medium for hierarchical object segmentation of large-scene remote sensing images based on multi-source pre-training model fusion decision-making, which can be used in infrastructure monitoring, natural resource management, environmental monitoring and other fields.

[0002] Background Field

[0003] Object segmentation of optical remote sensing images has far-reaching and multifaceted significance. It can clearly segment different types of objects from complex images, providing fine and accurate data support for land resource management, urban development research, disaster monitoring and emergency response. In recent years, pre-training of deep learning models can mine the potential features of massive data and enhance the model's ability to fit data. However, most of the current pre-training is for natural images, which is difficult to be effectively applied to remote sensing scenarios.

[0004] Natural images tend to focus on scenes, people, objects, etc. in daily life, while optical remote sensing images pay more attention to information on geography, environment, land use, etc. The emphasis and feature performance of the two are quite different. Therefore, there is still a performance bottleneck in directly using natural image pre-training models in remote sensing image vision tasks. In addition, existing remote sensing images are difficult to obtain, the cost of label acquisition is high, and the scenes of remote sensing images have certain particularities. Direct remote sensing image pre-training will face the defect of limited model generalization performance. At present, more and more scholars in related fields are focusing on how to use multi-source data to provide complementary information for the model to improve the accuracy and reliability of the model and enhance the generalization ability of the model. At the same time, by introducing other relevant data sources, the problem of insufficient single data volume can also be compensated.

[0005] The patent application document with publication number CN118314353A discloses a remote sensing image segmentation method based on dual-branch multi-scale feature fusion, which solves the problem that existing remote sensing image segmentation methods cannot effectively process ground features of different scales, achieve accurate segmentation without consuming a lot of calculations and memory, and have unstable image segmentation results. However, although this method expands the remote sensing image source at the scale level, it only extracts features on remote sensing images, resulting in poor generalization ability and fault tolerance of the model when facing complex scenes. At the same time, this method has high requirements on data consistency and synchronization, and requires obtaining remote sensing images of different scales in the same area. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a method, system, equipment and medium for hierarchical object segmentation of large-scene remote sensing images based on multi-source pre-trained model fusion decision, which combines the general capabilities of the natural image pre-training model and the proprietary capabilities of the remote sensing image pre-training model through the fusion decision method, so as to solve the problems of low accuracy of large-scene remote sensing image object segmentation categories and high requirements on data consistency and synchronization in the prior art.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A hierarchical object segmentation method for large-scene remote sensing images based on multi-source pre-training model fusion decision-making includes the following steps:

[0009] Step 1: Build a remote sensing image pre-training dataset D pre : Preprocess and crop massive optical remote sensing images to obtain a remote sensing image pre-training dataset D pre ;

[0010] Step 2: Construct a remote sensing image fine-tuning dataset: First, construct a set of typical ground object categories in a large scene and merge the optical remote sensing images with labels; second, perform non-overlapping cropping on the merged optical remote sensing images and their labels to obtain a remote sensing image fine-tuning dataset, and divide it into training set, validation set, and test set according to the proportion;

[0011] Step 3: Model pre-training: Use the remote sensing image pre-training dataset D obtained in step 1 pre , and use the mask reconstruction strategy to train the pre-trained model of the encoder-decoder structure to obtain the remote sensing image pre-training weights;

[0012] Step 4: Model fine-tuning and inference: First, the encoder-semantic segmentation head structure model is fine-tuned based on the remote sensing image pre-training weights obtained in step 3 and the natural image pre-training weights obtained by pre-training the ImageNet dataset, using the training set and validation set generated in step 2 to obtain two semantic segmentation models with the same structure but different weights; secondly, the two semantic segmentation models are used to test the test set generated in step 2 to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model;

[0013] Step 5: Rule-based fusion decision: Based on the set rules, the prediction results of the natural image pre-training model and the remote sensing image pre-training model in step 4 are fused and decided to obtain the remote sensing image object segmentation result;

[0014] Step 6: Performance evaluation: Calculate the evaluation indicators of each category and overall quality in the remote sensing image segmentation results obtained in step 5 to quantify the fusion performance.

[0015] The step 1 constructs a remote sensing image pre-training dataset D pre The specific process is:

[0016] Step 1.1: Perform preprocessing operations such as data screening and data cleaning on massive optical remote sensing images to obtain high-quality remote sensing image data;

[0017] Step 1.2: Perform non-overlapping cropping of the high-quality remote sensing image data obtained in step 1.1 according to the size of 512*512 to obtain the remote sensing image pre-training dataset D pre .

[0018] The specific process of constructing the fine-tuning dataset in step 2 is as follows:

[0019] Step 2.1: Construct a hierarchical classification system of typical features in the large scene. The first-level category includes but is not limited to: cultivated land, forest and grass cover, houses and buildings, transportation, structures, artificial excavation land, exposed surface, and water area; the second-level category includes: terraces, paddy fields, dry land, orchards, tea gardens, wetlands, rural villages, educational land, railways, highways, other roads, other transportation facilities, silt dams, city walls (ancient towns), greenhouses, chemical plants, solar panels, open-pit metal mines, river sand mining plants, gullies, other exposed surfaces, rivers, lakes, trees, shrubs, artificial grasslands, natural grasslands, high-rise office buildings, factory areas, residential areas, transfer reservoirs, aquaculture water surfaces, and planting water surfaces;

[0020] Step 2.2: Select the labeled optical remote sensing images, merge the categories according to the category system constructed in step 2.1, and perform non-overlapping cropping according to the size of 512*512 to obtain the remote sensing image fine-tuning dataset D seg , and divided into training set, validation set and test set in the ratio of 8:1:1.

[0021] The specific process of the model pre-training in step 3 is as follows:

[0022] The pre-trained model of the encoder-decoder structure includes an encoder E and a decoder D;

[0023] Step 3.1: Model input: Use the remote sensing image pre-training dataset constructed in step 1.2 The masked sample D in pre As input to a pre-trained model of an encoder-decoder structure:

[0024]

[0025] Among them, Ψ(·) represents the pre-trained model, and θ represents the optimization parameters in the model;

[0026] Step 3.2: Define encoder E: The backbone network used by encoder E is Swin-S. The Swin-S network consists of 4 stages. The number of Swin Transformer blocks contained in each stage from front to back is 2, 2, 18, and 2 respectively:

[0027] F pre =Encoder(D pre )

[0028] Among them, Encoder(·) represents the encoder network, F pre Represents the characteristics of the encoder output;

[0029] Step 3.3: Define decoder D: Decoder D uses a fully connected network with the same output as sample D pre Reconstruction results with the same dimensions:

[0030]

[0031] Among them, Decoder(·) represents the decoder network, Represents the reconstruction result;

[0032] Step 3.4: Design a mask reconstruction strategy: select a mask block size and a mask rate, and use the mask block of the selected size to randomly mask part of the input data according to the mask rate;

[0033] Step 3.5: Construct the reconstruction loss and use the mean absolute error as the loss function to calculate the difference between the original image and the reconstructed image:

[0034]

[0035] Among them, L 1 (·) represents the mean absolute error, y is the true value, is the predicted value, n is the number of samples;

[0036] Step 3.6: Pre-train the pre-trained model and save the encoder weights after training as remote sensing image pre-training weights.

[0037] The specific process of model fine-tuning and reasoning in step 4 is as follows:

[0038] The model of the encoder-semantic segmentation head structure includes an encoder E and a semantic segmentation head H;

[0039] Step 4.1: Define the semantic segmentation head H: Use 1×1 convolution to reduce the channel dimension of the input shallow features, that is, the output of the first stage of encoder E; use atrous spatial convolutional pyramid pooling (ASPP) to extract multi-scale deep features of the input, that is, the output of the fourth stage of encoder E, and finally concatenate the processed shallow features and deep features and input them into the classifier to obtain the final output;

[0040] Step 4.2: Connect the encoder E in step 3.2 to the semantic segmentation head H, and use the output of encoder E stage 1 as the shallow feature input and the output of stage 4 as the deep feature input;

[0041] Step 4.3: Use the remote sensing image pre-training weights obtained in step 3 and the natural image pre-training weights obtained by pre-training the ImageNet dataset as the initial weights of encoder E, and use the remote sensing image fine-tuning dataset in step 2.2 for fine-tuning to obtain two semantic segmentation models with the same structure but different weights;

[0042] Step 4.4: Separately assign the samples D in the test set generated in step 2.2 to seg Input the two semantic segmentation models fine-tuned in step 4.3 to predict the results and obtain the predicted category probability distribution Pro out And the prediction result Pred out :

[0043]

[0044] in, represents the semantic segmentation head network, softmax(·) represents the normalized exponential function, and argmax(·) represents the maximum independent variable value function.

[0045] The specific process of rule-based fusion decision in step 5 is as follows:

[0046] Step 5.1: Calculate the maximum probability at pixel level: Probability distribution of the category obtained in step 4.4 out Calculate the maximum probability for each pixel in and get the maximum probability of each pixel:

[0047]

[0048] Among them, Pro max represents the maximum probability of the pixel, and max(·) represents the maximum value calculation function;

[0049] Step 5.2: Construct pixel comparison rules and category injection: Based on the maximum probability of the pixel obtained in step 5.1, perform pixel-level probability comparison. If the natural image at this pixel is greater than the maximum probability of the remote sensing image under the set rules, the prediction of the natural image pre-training model is injected into the pixel as the result; otherwise, the prediction of the remote sensing image pre-training model is injected into the pixel as the result:

[0050]

[0051] Among them, Pred fusion Represents the prediction result after fusion, Pred pre Represents the prediction result of the remote sensing image pre-training model, Pred ImageNet represents the prediction result of the natural image pre-training model, It represents the maximum probability of the remote sensing image pre-training model at this pixel. represents the maximum probability of the natural image pre-training model at this pixel, and ρ represents the rule threshold.

[0052] The evaluation indicators for each category in step 6 include:

[0053] Precision:

[0054]

[0055] Among them, TP stands for True Positive (the number of samples correctly predicted as positive examples), and FP stands for False Positive (the number of samples incorrectly predicted as positive examples);

[0056] Intersection over Union (IoU):

[0057]

[0058] Among them, TP represents true positive (the number of samples correctly predicted as positive examples), FP represents false positive (the number of samples incorrectly predicted as positive examples), and FN represents false negative (the number of samples incorrectly predicted as negative examples).

[0059] Recall:

[0060]

[0061] Among them, TP represents true positive (the number of samples correctly predicted as positive examples), and FN represents false negative (the number of samples incorrectly predicted as negative examples);

[0062] F1score value:

[0063]

[0064] The overall quality evaluation indicators include:

[0065] Average intersection over union (MIoU):

[0066]

[0067] Among them, k represents the number of categories, p ij It represents the number of pixels that originally belonged to the i-th class but were predicted to be the j-th class.

[0068] The present invention also provides a large-scene remote sensing image hierarchical object segmentation system based on multi-source pre-training model fusion decision-making, comprising:

[0069] The remote sensing image pre-training dataset construction module is used to pre-process and crop massive optical remote sensing images to obtain the remote sensing image pre-training dataset D pre ;

[0070] The remote sensing image fine-tuning dataset construction module is used to construct a set of typical ground object categories in a large scene and merge the optical remote sensing images with labels; non-overlapping cropping of the merged optical remote sensing images and their labels is performed to obtain the remote sensing image fine-tuning dataset, and the training set, validation set, and test set are divided according to the proportion;

[0071] Model pre-training module, used to use remote sensing image pre-training dataset D pre , and use the mask reconstruction strategy to train the pre-trained model of the encoder-decoder structure to obtain the remote sensing image pre-training weights;

[0072] The model fine-tuning and inference module is used to fine-tune the model of the encoder-semantic segmentation head structure based on the remote sensing image pre-training weights and the natural image pre-training weights obtained by pre-training the ImageNet dataset, using the training set and the validation set to obtain two semantic segmentation models with the same structure but different weights; the two semantic segmentation models are used to test the test set respectively to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model;

[0073] The remote sensing image object segmentation module is used to make a fusion decision on the prediction results of the natural image pre-training model and the remote sensing image pre-training model based on the set rules to obtain the remote sensing image object segmentation results;

[0074] The performance evaluation module is used to calculate the evaluation indicators of each category and overall quality in the remote sensing image segmentation results and quantify the fusion performance.

[0075] The present invention also provides a large-scene remote sensing image hierarchical object segmentation device based on multi-source pre-training model fusion decision, including:

[0076] Memory: a computer program for storing the aforementioned method for hierarchical object segmentation of large-scene remote sensing images based on multi-source pre-training model fusion decision, which is a computer-readable device;

[0077] Processor: used to implement the method for hierarchical object segmentation of large-scene remote sensing images based on multi-source pre-training model fusion decision when executing the computer program.

[0078] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for hierarchical object segmentation of large-scene remote sensing images based on multi-source pre-training model fusion decision.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] 1. The present invention builds a model through multi-source data pre-training, that is, feature extraction and image reconstruction are performed on natural images and remote sensing images respectively, which can combine the general capabilities of the natural image pre-training model with the specific capabilities of the remote sensing image pre-training model to enhance the generalization of the model in different remote sensing scenarios.

[0081] 2. The present invention compares the predicted category probabilities of the natural image pre-training model and the remote sensing image pre-training model at the same pixel through rule-based fusion decision-making. When the category predicted by the natural image pre-training model is different from the category predicted by the remote sensing image pre-training model, the more credible category is taken as the prediction result, which can effectively remove noise points and improve the abnormal situation in the existing technology of large-scene remote sensing object segmentation results.

[0082] In summary, the present invention has the advantages of high segmentation accuracy and strong robustness through multi-source data pre-training model modeling and rule-based fusion decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a flow chart of the method of the present invention.

[0084] Figure 2 This is a graph of experimental prediction results of the present invention compared with the natural image pre-training model and the remote sensing image pre-training model on a test set with an image size of 512×512.

[0085] Figure 3 This is a graph of experimental prediction results of the present invention comparing high-resolution remote sensing images of complex scenes in Guanzhong and southern Shaanxi with a natural image pre-training model and a remote sensing image pre-training model. DETAILED DESCRIPTION

[0086] The present invention is further described below in conjunction with the accompanying drawings.

[0087] The purpose of the present invention is to overcome the defects of the prior art. In view of the poor scene adaptability of natural image pre-training models and the inability to accurately identify complex objects, and the insufficient generalization ability of remote sensing image pre-training models, a large-scene remote sensing image hierarchical object segmentation method based on multi-source pre-training model fusion decision is proposed to solve the problem of low accuracy of the prior art in large scenes.

[0088] like Figure 1 As shown, a hierarchical object segmentation method for large-scene remote sensing images based on multi-source pre-training model fusion decision-making includes the following steps:

[0089] Step 1: Build a remote sensing image pre-training dataset D pre : Preprocess and crop massive optical remote sensing images to obtain a remote sensing image pre-training dataset D pre ;

[0090] The step 1 constructs a remote sensing image pre-training dataset D pre The specific process is:

[0091] Step 1.1: Perform data screening and data cleaning preprocessing operations on massive optical remote sensing images of Shaanxi Province to obtain high-quality remote sensing image data;

[0092] Step 1.2: Perform non-overlapping cropping of the high-quality remote sensing image data obtained in step 1.1 according to the size of 515*512 to obtain a pre-training dataset D containing a certain number of remote sensing images. pre .

[0093] Step 2: Construct a remote sensing image fine-tuning dataset: First, construct a set of typical ground object categories in a large scene, and merge the optical remote sensing images with labels into categories; second, perform non-overlapping cropping on the merged optical remote sensing images and their labels to obtain a remote sensing image fine-tuning dataset, and divide it into training set, validation set, and test set according to a certain ratio;

[0094] The specific process of constructing the fine-tuning dataset in step 2 is as follows:

[0095] Step 2.1: Construct a hierarchical classification system of typical features in the large scene. The first-level category includes but is not limited to: cultivated land, forest and grass cover, houses and buildings, transportation, structures, artificial excavation land, exposed surface, and water area; the second-level category includes: terraces, paddy fields, dry land, orchards, tea gardens, wetlands, rural villages, educational land, railways, highways, other roads, other transportation facilities, silt dams, city walls (ancient towns), greenhouses, chemical plants, solar panels, open-pit metal mines, river sand mining plants, gullies, other exposed surfaces, rivers, lakes, trees, shrubs, artificial grasslands, natural grasslands, high-rise office buildings, factory areas, residential areas, transfer reservoirs, aquaculture water surfaces, and planting water surfaces;

[0096] Step 2.2: Select the labeled optical remote sensing images of Shaanxi Province, merge the categories according to the category system constructed in step 2.1, and perform non-overlapping cropping according to the size of 512*512 to obtain a fine-tuning dataset D containing a certain number of remote sensing images. seg , and divided into training set, validation set and test set in the ratio of 8:1:1.

[0097] Step 3: Model pre-training: Use the remote sensing image pre-training dataset D obtained in step 1 pre , and use the mask reconstruction strategy to train the pre-trained model of the encoder-decoder structure to obtain the remote sensing image pre-training weights;

[0098] The specific process of the model pre-training in step 3 is as follows:

[0099] The pre-trained model of the encoder-decoder structure includes an encoder E and a decoder D;

[0100] Step 3.1: Model input: masked samples D from the remote sensing image pre-training dataset constructed in step 1.2 pre As input to a pre-trained model of an encoder-decoder structure:

[0101] argminE(Ψ(D pre )|θ)

[0102] θ

[0103] Here, Ψ(·) represents the pre-trained model and θ represents the optimized parameters in the model.

[0104] Step 3.2: Define encoder E: The backbone network used by encoder E in the present invention is Swin-S. The Swin-S network is mainly composed of 4 stages. The number of Swin Transformer blocks contained in each stage from front to back is 2, 2, 18, and 2 respectively:

[0105] F pre =Encoder(D pre)

[0106] Among them, Encoder(·) represents the encoder network, F pre Represents the characteristics of the encoder output;

[0107] Step 3.3: Define decoder D: Decoder D uses a fully connected network with the same output as D pre Reconstruction results with the same dimensions:

[0108]

[0109] Among them, Decoder(·) represents the decoder network, Represents the reconstruction result;

[0110] Step 3.4: Design a mask reconstruction strategy: select a mask block size and a mask rate, and use the mask block of the selected size to randomly mask part of the input data according to the mask rate;

[0111] Step 3.5: Construct reconstruction loss. The present invention uses mean absolute error as the loss function to calculate the difference between the original image and the reconstructed image:

[0112]

[0113] Among them, L 1 (·) represents the mean absolute error, y is the true value, is the predicted value and n is the number of samples.

[0114] Step 3.6: Pre-train the pre-trained model and save the encoder weights after training as remote sensing image pre-training weights.

[0115] Step 4: Model fine-tuning and inference: First, the encoder-semantic segmentation head structure model is fine-tuned based on the remote sensing image pre-training weights obtained in step 3 and the natural image pre-training weights obtained by pre-training the ImageNet dataset, using the training set and validation set generated in step 2 to obtain two semantic segmentation models with the same structure but different weights; secondly, the two semantic segmentation models are used to test the test set generated in step 2 to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model;

[0116] The specific process of model fine-tuning and reasoning in step 4 is as follows:

[0117] The model of the encoder-semantic segmentation head structure includes an encoder E and a semantic segmentation head H;

[0118] Step 4.1: Define the semantic segmentation head H: Use 1×1 convolution to reduce the channel dimension of the input shallow features (the output of the first stage of encoder E); use atrous spatial pyramid pooling (ASPP) to extract multi-scale deep features of the input (the output of the fourth stage of encoder E), and finally concatenate the processed shallow features and deep features and input them into the classifier to obtain the final output;

[0119] Step 4.2: Connect the encoder E in step 3.2 to the semantic segmentation head H, and use the output of encoder E stage 1 as the shallow feature input and the output of stage 4 as the deep feature input;

[0120] Step 4.3: Use the remote sensing image pre-training weights obtained in step 3 and the natural image pre-training weights obtained by pre-training the ImageNet dataset as the initial weights of encoder E, and use the remote sensing image fine-tuning dataset in step 2.2 for fine-tuning to obtain two semantic segmentation models with the same structure but different weights;

[0121] Step 4.4: Samples D in the remote sensing image test set generated in step 2.2 are seg Input the two semantic segmentation models fine-tuned in step 4.3 to predict the results and obtain the predicted category probability distribution Pro out And the prediction result Pred out :

[0122]

[0123] in, represents the semantic segmentation head network, softmax(·) represents the normalized exponential function, and argmax(·) represents the maximum independent variable value function.

[0124] Step 5: Rule-based fusion decision: Based on the set rules, the prediction results of the natural image pre-training model and the remote sensing image pre-training model in step 4 are fused and decided to obtain higher quality remote sensing image semantic segmentation results;

[0125] The rule-based fusion decision-making idea of ​​step 5 is:

[0126] Step 5.1: Calculate the maximum probability at pixel level: Probability distribution of the category obtained in step 4.4 out Calculate the maximum probability for each pixel in and get the maximum probability of each pixel:

[0127]

[0128] Among them, Pro maxrepresents the maximum probability of the pixel, and max(·) represents the maximum value calculation function;

[0129] Step 5.2: Construct pixel comparison rules and category injection: Based on the maximum probability of the pixel obtained in step 5.1, perform pixel-level probability comparison. If the natural image at this pixel is greater than the maximum probability of the remote sensing image under the set rules, the prediction of the natural image pre-training model is injected into the pixel as the result; otherwise, the prediction of the remote sensing image pre-training model is injected into the pixel as the result:

[0130]

[0131] Among them, Pred fusion Represents the prediction result after fusion, Pred pre Represents the prediction result of the remote sensing image pre-training model, Pred ImageNet represents the prediction result of the natural image pre-training model, It represents the maximum probability of the remote sensing image pre-training model at this pixel. represents the maximum probability of the natural image pre-training model at this pixel, ρ represents the rule threshold, and in the present invention, ρ=0.9.

[0132] Step 6: Performance evaluation: Calculate the evaluation indicators of each category and overall quality in the remote sensing image segmentation results obtained in step 5 to quantify the fusion performance.

[0133] The evaluation indicators for each category in step 6 include:

[0134] Precision:

[0135]

[0136] Among them, TP stands for True Positive (the number of samples correctly predicted as positive examples), and FP stands for False Positive (the number of samples incorrectly predicted as positive examples);

[0137] Intersection over Union (IoU):

[0138]

[0139] Among them, TP represents true positive (the number of samples correctly predicted as positive examples), FP represents false positive (the number of samples incorrectly predicted as positive examples), and FN represents false negative (the number of samples incorrectly predicted as negative examples).

[0140] Recall:

[0141]

[0142] Among them, TP represents true positive (the number of samples correctly predicted as positive examples), and FN represents false negative (the number of samples incorrectly predicted as negative examples);

[0143] F1score value:

[0144]

[0145] The indicators for overall quality evaluation include:

[0146] Average intersection over union (MIoU):

[0147]

[0148] Among them, k represents the number of categories, p ij It represents the number of pixels that originally belonged to the i-th class but were predicted to be the j-th class.

[0149] The technical problem to be solved by the present invention is that the existing technology usually uses a single-source pre-training model for single modeling, which results in that natural image pre-training cannot be directly applied to remote sensing image object segmentation tasks; at the same time, since remote sensing image pre-training is difficult to mine feature diversity in massive data, there will be obvious performance bottlenecks when pre-training remote sensing images.

[0150] In view of the above problems, the present invention proposes a hierarchical object segmentation method for large-scene remote sensing images based on multi-source pre-training model fusion decision. Through a novel rule-based fusion decision method, the abnormal situation in the object segmentation results of large-scene remote sensing images in the existing technology is greatly improved, and the robustness of the model is enhanced.

[0151] like Figure 2 As shown in the figure, the experimental prediction results of the present invention on the test set are compared with the natural image pre-training model and the remote sensing image pre-training model in typical scenarios; through comparison, it is found that the present invention has obvious advantages in the classification results of other typical scenarios such as roads, artificial grasslands, orchards, educational land, and factory areas. Figure 2 It can be seen from scene 1 that the natural image pre-training model has a poor effect in distinguishing between educational land and factory areas, and the remote sensing image pre-training model does not extract the artificial grassland category. The present invention uses fusion decision-making to make up for the shortcomings of the two models and form a more accurate classification result. Figure 2 It can be seen from scene 2 that the classification results of the natural image pre-training model and the remote sensing image pre-training model have noise points in local areas and incoherent classification results. The present invention effectively removes the noise points and has better noise suppression capability and robustness.

[0152] like Figure 3 As shown, this is a generalized prediction result diagram of the present invention compared with the natural image pre-training model and the remote sensing image pre-training model on the high-resolution remote sensing image data of complex scenes in Guanzhong and southern Shaanxi; wherein, scene 1 is a high-resolution remote sensing image of a complex scene in Guanzhong. By comparison, it is found that the present invention removes the noise points distributed in the urban and forest areas in the classification results of the natural image pre-training model and the remote sensing image pre-training model, and can accurately classify most categories in the urban area; scene 2 is a high-resolution remote sensing image of a complex scene in southern Shaanxi. By comparison, it is found that the present invention has a clearer boundary between trees and dry land, and removes some areas that were mistakenly identified as water bodies and backgrounds.

[0153] As shown in the quantitative results in Table 1 below, the overall mIoU (%) of the present invention is improved by 1.43 and 4.09 respectively compared with the natural image pre-training model and the remote sensing image pre-training model. At the same time, the present invention improves Precision (precision), Recall (recall rate) and IoU (intersection over union ratio) in most categories, especially "adjustment reservoir", "aquaculture water surface", "railway", "other roads", "other transportation facilities", "chemical plant", "factory area" and other categories. The improvement is obvious, and the IoU (%) of the natural image pre-training model is improved by 3.38, 2.21, 3.37, 2.72, 2.42, 3.91 and 2.02 respectively, and the IoU (%) of the remote sensing image pre-training model is improved by 3.16, 2.94, 5.94, 1.82, 2.18, 3.13 and 3.18 respectively. It can be seen that the present invention effectively improves the accuracy of large-scene remote sensing image segmentation categories.

[0154] Table 1 Comparison of various evaluation indicators of the present invention (bottom) with the natural image pre-training model (top) and the remote sensing image pre-training model (middle) in the same data set.

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161] The present invention also provides a large-scene remote sensing image hierarchical object segmentation system based on multi-source pre-training model fusion decision-making, comprising:

[0162] The remote sensing image pre-training dataset construction module is used to implement the pre-processing and cropping of massive optical remote sensing images in step 1 to obtain the remote sensing image pre-training dataset D pre ;

[0163] The remote sensing image fine-tuning dataset construction module is used to implement the typical ground object category set in the large scene in step 2, and to merge the optical remote sensing images with labels; to perform non-overlapping cropping on the optical remote sensing images and their labels after the category merger to obtain the remote sensing image fine-tuning dataset, and to divide it into training set, validation set and test set according to the proportion;

[0164] The model pre-training module is used to implement the remote sensing image pre-training dataset D obtained in step 1 in step 3. pre , and use the mask reconstruction strategy to train the pre-trained model of the encoder-decoder structure to obtain the remote sensing image pre-training weights;

[0165] The model fine-tuning and inference module is used to implement the model of the encoder-semantic segmentation head structure in step 4, based on the remote sensing image pre-training weights obtained in step 3 and the natural image pre-training weights obtained by pre-training the ImageNet dataset, and fine-tune the model using the training set and validation set generated in step 2 to obtain two semantic segmentation models with the same structure but different weights; the test set generated in step 2 is tested using the two semantic segmentation models to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model;

[0166] The remote sensing image object segmentation module is used to implement the set rules in step 5, make a fusion decision on the prediction results of the natural image pre-training model and the remote sensing image pre-training model in step 4, and obtain the remote sensing image object segmentation result;

[0167] The performance evaluation module is used to implement the evaluation indicators of each category and overall quality in the remote sensing image segmentation result obtained in step 5 calculated in step 6, and quantify the fusion performance.

[0168] The present invention also provides a large-scene remote sensing image hierarchical object segmentation device based on multi-source pre-training model fusion decision, including:

[0169] Memory: a computer program for storing the aforementioned method for hierarchical object segmentation of large-scene remote sensing images based on multi-source pre-training model fusion decision, which is a computer-readable device;

[0170] Processor: used to implement the method for hierarchical object segmentation of large-scene remote sensing images based on multi-source pre-training model fusion decision when executing the computer program.

[0171] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for hierarchical object segmentation of large-scene remote sensing images based on multi-source pre-training model fusion decision.

Claims

1. A hierarchical object segmentation method for large-scene remote sensing images based on multi-source pre-training model fusion decision, characterized in that: The following steps are involved: Step 1: Build a remote sensing image pre-training dataset D pre : Preprocess and crop massive optical remote sensing images to obtain a remote sensing image pre-training dataset D pre ; Step 2: Construct a remote sensing image fine-tuning dataset: First, construct a set of typical ground object categories in a large scene and merge the optical remote sensing images with labels; second, perform non-overlapping cropping on the merged optical remote sensing images and their labels to obtain a remote sensing image fine-tuning dataset, and divide it into training set, validation set, and test set according to the proportion; Step 3: Model pre-training: Use the remote sensing image pre-training dataset D obtained in step 1 pre , and use the mask reconstruction strategy to train the pre-trained model of the encoder-decoder structure to obtain the remote sensing image pre-training weights; Step 4: Model fine-tuning and inference: First, the encoder-semantic segmentation head structure model is fine-tuned based on the remote sensing image pre-training weights obtained in step 3 and the natural image pre-training weights obtained by pre-training the ImageNet dataset, using the training set and validation set generated in step 2 to obtain two semantic segmentation models with the same structure but different weights; secondly, the two semantic segmentation models are used to test the test set generated in step 2 to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model; Step 5: Rule-based fusion decision: Based on the set rules, the prediction results of the natural image pre-training model and the remote sensing image pre-training model in step 4 are fused and decided to obtain the remote sensing image object segmentation result; Step 6: Performance evaluation: Calculate the evaluation indicators of each category and overall quality in the remote sensing image segmentation results obtained in step 5 to quantify the fusion performance.

2. According to claim 1, a large-scene remote sensing image hierarchical object segmentation method based on multi-source pre-training model fusion decision-making is characterized in that: The step 1 constructs a remote sensing image pre-training dataset D pre The specific process is: Step 1.1: Perform preprocessing operations such as data screening and data cleaning on massive optical remote sensing images to obtain high-quality remote sensing image data; Step 1.2: Perform non-overlapping cropping of the high-quality remote sensing image data obtained in step 1.1 according to the size of 512*512 to obtain the remote sensing image pre-training dataset D pre .

3. According to claim 1, a large-scene remote sensing image hierarchical object segmentation method based on multi-source pre-training model fusion decision-making is characterized in that: The specific process of constructing the fine-tuning dataset in step 2 is as follows: Step 2.1: Construct a hierarchical classification system of typical features in the large scene. The first-level category includes but is not limited to: cultivated land, forest and grass cover, houses and buildings, transportation, structures, artificial excavation land, exposed surface, and water area; the second-level category includes: terraces, paddy fields, dry land, orchards, tea gardens, wetlands, rural villages, educational land, railways, highways, other roads, other transportation facilities, silt dams, city walls (ancient towns), greenhouses, chemical plants, solar panels, open-pit metal mines, river sand mining plants, gullies, other exposed surfaces, rivers, lakes, trees, shrubs, artificial grasslands, natural grasslands, high-rise office buildings, factory areas, residential areas, transfer reservoirs, aquaculture water surfaces, and planting water surfaces; Step 2.2: Select the labeled optical remote sensing images, merge the categories according to the category system constructed in step 2.1, and perform non-overlapping cropping according to the size of 512*512 to obtain the remote sensing image fine-tuning dataset D seg , and divided into training set, validation set and test set in the ratio of 8:1:

1.

4. According to claim 1, a large-scene remote sensing image hierarchical object segmentation method based on multi-source pre-training model fusion decision-making is characterized in that: The specific process of the model pre-training in step 3 is as follows: The pre-trained model of the encoder-decoder structure includes an encoder E and a decoder D; Step 3.1: Model input: Use the remote sensing image pre-training dataset D constructed in step 1.2 pre The masked sample D in pre As input to a pre-trained model of an encoder-decoder structure: Among them, Ψ(·) represents the pre-trained model, and θ represents the optimization parameters in the model; Step 3.2: Define encoder E: The backbone network used by encoder E is Swin-S. The Swin-S network consists of 4 stages. The number of Swin Transformer blocks contained in each stage from front to back is 2, 2, 18, and 2 respectively: F pre =Encoder(D pre ) Among them, Encoder(·) represents the encoder network, F pre Represents the characteristics of the encoder output; Step 3.3: Define decoder D: Decoder D uses a fully connected network with the same output as sample D pre Reconstruction results with the same dimensions: Among them, Decoder(·) represents the decoder network, Represents the reconstruction result; Step 3.4: Design a mask reconstruction strategy: select a mask block size and a mask rate, and use the mask block of the selected size to randomly mask part of the input data according to the mask rate; Step 3.5: Construct the reconstruction loss and use the mean absolute error as the loss function to calculate the difference between the original image and the reconstructed image: Where L1(·) represents the mean absolute error, y is the true value, is the predicted value, n is the number of samples; Step 3.6: Pre-train the pre-trained model and save the encoder weights after training as remote sensing image pre-training weights.

5. According to claim 1, a large-scene remote sensing image hierarchical object segmentation method based on multi-source pre-training model fusion decision-making is characterized in that: The specific process of model fine-tuning and reasoning in step 4 is as follows: The model of the encoder-semantic segmentation head structure includes an encoder E and a semantic segmentation head H; Step 4.1: Define the semantic segmentation head H: Use 1×1 convolution to reduce the channel dimension of the input shallow features, that is, the output of the first stage of encoder E; use atrous spatial pyramid pooling (ASPP) to extract multi-scale deep features of the input, that is, the output of the fourth stage of encoder E, and finally concatenate the processed shallow features and deep features and input them into the classifier to obtain the final output; Step 4.2: Connect the encoder E in step 3.2 to the semantic segmentation head H, and use the output of encoder E stage 1 as the shallow feature input and the output of stage 4 as the deep feature input; Step 4.3: Use the remote sensing image pre-training weights obtained in step 3 and the natural image pre-training weights obtained by pre-training the ImageNet dataset as the initial weights of encoder E, and use the remote sensing image fine-tuning dataset in step 2.2 for fine-tuning to obtain two semantic segmentation models with the same structure but different weights; Step 4.4: Separately assign the samples D in the test set generated in step 2.2 to seg Input the two semantic segmentation models fine-tuned in step 4.3 to predict the results and obtain the predicted category probability distribution Pro out And the prediction result Pred out : Pro out =softmax(H(Encoder(D seg ))) Pred out =argmax(H(Encoder(D seg ))) Among them, H(·) represents the semantic segmentation head network, softmax(·) represents the normalized exponential function, and argmax(·) represents the maximum independent variable value function.

6. According to claim 1, a large-scene remote sensing image hierarchical object segmentation method based on multi-source pre-training model fusion decision-making is characterized in that: The specific process of rule-based fusion decision in step 5 is as follows: Step 5.1: Calculate the maximum probability at pixel level: Probability distribution of the category obtained in step 4.4 out Calculate the maximum probability for each pixel in and get the maximum probability of each pixel: Among them, Pro max represents the maximum probability of the pixel, and max(·) represents the maximum value calculation function; Step 5.2: Construct pixel comparison rules and category injection: Based on the maximum probability of the pixel obtained in step 5.1, perform pixel-level probability comparison. If the natural image at this pixel is greater than the maximum probability of the remote sensing image under the set rules, the prediction of the natural image pre-training model is injected into the pixel as the result; otherwise, the prediction of the remote sensing image pre-training model is injected into the pixel as the result: Among them, Pred fusion Represents the prediction result after fusion, Pred pre Represents the prediction result of the remote sensing image pre-training model, Pred ImageNet represents the prediction result of the natural image pre-training model, It represents the maximum probability of the remote sensing image pre-training model at this pixel. represents the maximum probability of the natural image pre-training model at this pixel, and ρ represents the rule threshold.

7. According to claim 1, a large-scene remote sensing image hierarchical object segmentation method based on multi-source pre-training model fusion decision-making is characterized in that: The evaluation indicators for each category in step 6 include: Precision: Among them, TP stands for True Positive (the number of samples correctly predicted as positive examples), and FP stands for False Positive (the number of samples incorrectly predicted as positive examples); Intersection over Union (IoU): Among them, TP represents true positive (the number of samples correctly predicted as positive examples), FP represents false positive (the number of samples incorrectly predicted as positive examples), and FN represents false negative (the number of samples incorrectly predicted as negative examples). Recall: Among them, TP represents true positive (the number of samples correctly predicted as positive examples), and FN represents false negative (the number of samples incorrectly predicted as negative examples); F1score value: The overall quality evaluation indicators include: Average intersection over union (MIoU): Among them, k represents the number of categories, p ij It represents the number of pixels that originally belonged to the i-th class but were predicted to be the j-th class.

8. A large-scene remote sensing image hierarchical object segmentation system based on multi-source pre-training model fusion decision-making based on the method described in any one of claims 1 to 7, characterized in that: include: The remote sensing image pre-training dataset construction module is used to pre-process and crop massive optical remote sensing images to obtain a remote sensing image pre-training dataset The remote sensing image fine-tuning dataset construction module is used to construct a set of typical ground object categories in a large scene and merge the optical remote sensing images with labels; non-overlapping cropping of the merged optical remote sensing images and their labels is performed to obtain the remote sensing image fine-tuning dataset, and the training set, validation set, and test set are divided according to the proportion; Model pre-training module, used to use remote sensing image pre-training dataset D pre , and use the mask reconstruction strategy to train the pre-trained model of the encoder-decoder structure to obtain the remote sensing image pre-training weights; The model fine-tuning and inference module is used to fine-tune the model of the encoder-semantic segmentation head structure based on the remote sensing image pre-training weights and the natural image pre-training weights obtained by pre-training the ImageNet dataset, using the training set and the validation set to obtain two semantic segmentation models with the same structure but different weights; the two semantic segmentation models are used to test the test set respectively to obtain the prediction results of the natural image pre-training model and the remote sensing image pre-training model; The remote sensing image object segmentation module is used to make a fusion decision on the prediction results of the natural image pre-training model and the remote sensing image pre-training model based on the set rules to obtain the remote sensing image object segmentation results; The performance evaluation module is used to calculate the evaluation indicators of each category and overall quality in the remote sensing image segmentation results and quantify the fusion performance.

9. A hierarchical object segmentation device for large-scene remote sensing images based on multi-source pre-training model fusion decision, characterized in that: include: Memory: a computer program for storing a method for hierarchical object segmentation of a large-scene remote sensing image based on multi-source pre-training model fusion decision-making as described in any one of claims 1 to 7, which is a computer-readable device; Processor: used to implement the large-scene remote sensing image hierarchical object segmentation method based on multi-source pre-training model fusion decision as described in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a large-scene remote sensing image hierarchical object segmentation method based on multi-source pre-training model fusion decision as described in any one of claims 1 to 7.

Images