Cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison

Abstract

The invention discloses a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison, and the method comprises the steps: inputting a target domain image into a cross-domain remote sensing image semantic segmentation model, and outputting a confidence coefficient; the step of training the cross-domain remote sensing image semantic segmentation model comprises the steps of preprocessing a source domain image with a label and a target domain image without a label; mixing the preprocessed source domain image and the target domain image through an adaptive mixing strategy to obtain a mixed image; semantic features of the source domain image and the target domain image are extracted, and the semantic features of the source domain and the target domain are aligned through prototype contrast learning; and training a cross-domain remote sensing image semantic segmentation model based on a semantic segmentation framework Segform by using the mixed image, the source domain image and the confidence coefficient. According to the method, information interaction between the source domain and the target domain is effectively utilized through the hybrid strategy, and the model is better helped to learn semantic features of the same category between different domains.

Description

Technical Field

[0001] The invention relates to a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison, and belongs to the technical field of image semantic segmentation. Background Art

[0002] In recent years, the application scope of remote sensing images has become increasingly wide, and semantic segmentation as a basic task has also received more and more attention. For example, building extraction, forest coverage area calculation, and road segmentation are all tasks with practical significance. Although deep learning has made great progress in computer vision tasks and semantic segmentation technology has continued to mature, the effect of applying the model trained on the source domain image to the target domain image has significantly decreased, and cross-domain semantic segmentation has become a mainstream topic.

[0003] The cross-domain phenomenon in the field of remote sensing images is quite obvious. Because images captured by different regions, different times, and even different sensors may have significant differences. At the same time, in order to train an effective semantic segmentation model, a large amount of labeled data is required. This data includes not only the image itself, but also the category label of each pixel. However, obtaining and labeling this data is usually costly and time-consuming. Therefore, the model trained on the source domain image needs to be able to adapt to changes in image differences to maintain its generalization ability. The main problems faced by cross-domain remote sensing image semantic segmentation are as follows:

[0004] (1) The distribution of remote sensing images is very different, and the image category imbalance is even worse. Images are affected by various factors, resulting in more irrelevant image noise. For the same category, the features extracted by the model have significant differences, making it difficult to accurately identify features between different domains.

[0005] (2) The pseudo-labels of self-training usually contain noise, where the noise usually refers to incorrect labels. However, since the weights of the teacher model are iteratively updated by the student model, the noise problem of the pseudo-labels has a serious impact on the model, which will damage the stability and accuracy of the model training. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the defects of the prior art, provide a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison, and propose corresponding solutions to the problems of obvious distribution differences and pseudo-label noise.

[0007] Preferably, the present invention provides a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison, comprising: inputting a pre-acquired target domain image into a trained cross-domain remote sensing image semantic segmentation model, and outputting a predicted confidence;

[0008] Among them, the cross-domain remote sensing image semantic segmentation model is trained, including:

[0009] Step 1, obtaining a labeled cross-domain remote sensing image semantic segmentation source domain image and an unlabeled cross-domain remote sensing image semantic segmentation target domain image, and preprocessing the original images in the labeled cross-domain remote sensing image semantic segmentation source domain image and the unlabeled cross-domain remote sensing image semantic segmentation target domain image;

[0010] Step 2: Build a cross-domain remote sensing image semantic segmentation model based on the semantic segmentation framework Segformer;

[0011] Step 3, mixing the preprocessed cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image through an adaptive mixing strategy to obtain a mixed image;

[0012] Step 4: extract semantic features of the source domain image and the target domain image of the cross-domain remote sensing image semantic segmentation, and align the semantic features of the source domain and the target domain through prototype comparison learning;

[0013] Step 5, use the mixed image, cross-domain remote sensing image semantic segmentation source domain image and confidence to train the cross-domain remote sensing image semantic segmentation model, construct a mapping relationship between the mixed image as input, the cross-domain remote sensing image semantic segmentation source domain image as input and the confidence as output, and obtain a trained cross-domain remote sensing image semantic segmentation model.

[0014] Preferably, step 1 comprises:

[0015] Step (1.1), obtaining a labeled cross-domain remote sensing image semantic segmentation source domain image and an unlabeled cross-domain remote sensing image semantic segmentation target domain image, wherein the cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image both include the original image;

[0016] Step (1.2), cropping the original image, cutting the original image into images of a preset size.

[0017] Prioritize, step 2, build a cross-domain remote sensing image semantic segmentation model based on the semantic segmentation framework Segformer, including:

[0018] Step (2.1), build the semantic segmentation framework Segformer, which consists of two parts: encoder and decoder, to obtain the student model;

[0019] Step (2.2), obtain the teacher model based on the student model, and the parameter update formula of the teacher model is as follows:

[0020]

[0021] Among them, t represents the current iteration round, α represents the update weight, represents the parameters of the teacher model at the tth iteration, Represents the parameters of the student model at the tth iteration.

[0022] Preferably, step 3, mixing the preprocessed cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image through an adaptive mixing strategy to obtain a mixed image, including:

[0023] Step (3.1), send the cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image into the student model to obtain the category prediction confidence The cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image are sent to the teacher model to obtain the corresponding category prediction confidence

[0024] Step (3.2), calculate the accuracy value of the category prediction confidence:

[0025]

[0026] in, Represents the accuracy of the source domain image category pixels, Represents the accuracy of target domain image category pixels;

[0027] Step (3.3), calculate the proportionality coefficient r1:

[0028]

[0029] Step (3.4), calculate the minimum number of category pixels n of the source domain image for cross-domain remote sensing image semantic segmentation based on the label min and the maximum number of category pixels n in the source domain image of cross-domain remote sensing image semantic segmentation max , calculate the number of pixels in each category n c The proportional coefficient r2 in the source domain image corresponding to the semantic segmentation of the cross-domain remote sensing image is as follows:

[0030]

[0031] Step (3.5), multiply the scale factor r1 by the scale factor r2 to obtain the final source domain category pixel ratio r;

[0032] Compare the final source domain category pixel ratio r with 1. If the final source domain category pixel ratio r>1, then r=1; if the final source domain category pixel ratio r≤1, then the final source domain category pixel ratio r retains the original value;

[0033] Step (3.6), the accuracy of all source domain image category pixels Sort in ascending order, using the final source domain category pixel ratio r and the number of pixels in each category in the source domain n c Multiply them to get the number of source domain category pixels in the mixed image The accuracy of pixels from each source domain image category sorted in ascending order Select the corresponding quantity Create a mask matrix of the same size as the source domain image for cross-domain remote sensing image semantic segmentation Mask Matrix The values ​​of the positions corresponding to the source domain images of the cross-domain remote sensing image semantic segmentation are set to 1, and the rest are 0;

[0034] Step (3.7), loop step (3.1) to step (3.6), and obtain the mixed mask matrix of each category of the source domain image for all cross-domain remote sensing image semantic segmentation The mask matrix corresponding to the target domain image is obtained by semantic segmentation of cross-domain remote sensing images Obtain the mask matrix corresponding to the target domain image of the cross-domain remote sensing image semantic segmentation; multiply the target domain image mask matrix and the source domain image mask matrix with the corresponding target domain image and source domain image, and then add them to obtain a mixed image; multiply the target domain image mask matrix and the source domain image mask matrix with the corresponding target domain pseudo label and source domain real label, and then add them to obtain the label of the mixed image.

[0035] Prior to this, step 4 aligns the semantic features of the source domain and the target domain through prototype comparison learning, including:

[0036] Step (4.1), input the source domain image of the cross-domain remote sensing image semantic segmentation into the student model to obtain the source domain image feature maps of four stages, perform feature fusion on the source domain image feature maps of the four stages, and obtain the source domain feature map; input the target domain image of the cross-domain remote sensing image semantic segmentation into the teacher model to obtain the target domain image feature maps of four stages, perform feature fusion on the target domain image feature maps of the four stages, and obtain the target domain feature map;

[0037] Step (4.2), input the target domain image of the cross-domain remote sensing image semantic segmentation into the teacher model to obtain the pixel-level category confidence score; extract the predicted highest category confidence score from the obtained pixel-level category confidence scores and the second highest class confidence score

[0038] Step (4.3), calculate the reliability score of each category pixel

[0039]

[0040] Step (4.4), according to the reliability score Compute the prototype features of the target domain:

[0041]

[0042] In the formula, is the pixel-level semantic feature of each category in the target domain, N c is the number of pixels for each category;

[0043] Step (4.5) obtains the prototype features of each category in the target domain Finally, according to the label of the source domain image of the cross-domain remote sensing image semantic segmentation, a feature matrix with the same size as the source domain image of the cross-domain remote sensing image semantic segmentation is created;

[0044] Step (4.6), use cosine similarity to compare the feature matrix and source domain pixel features of the same size as the source domain image of the cross-domain remote sensing image semantic segmentation, and obtain the cosine similarity score sim c :

[0045]

[0046] is the source domain category pixel feature;

[0047] In step (4.7), the cosine similarity score is exponentially normalized; considering the pseudo-label noise problem, the entropy of the source domain pixels is used to calculate the prototype contrast loss L cs :

[0048]

[0049] Among them, k is the number of categories, N c is the number of pixels for each category, is the pixel-level semantic feature of each category in the target domain, sim c is the cosine similarity score.

[0050] Preferably, step (4.1) comprises:

[0051] Step (4.11), taking the width and height of the first source domain image feature map in the four stages as the standard, adjusting the width and height of the second source domain image feature map, the third source domain image feature map, and the fourth source domain image feature map to the same size as the first source domain image feature map; taking the width and height of the first target domain image feature map in the four stages as the standard, adjusting the width and height of the second target domain image feature map, the third target domain image feature map, and the fourth target domain image feature map to the same size as the first target domain image feature map;

[0052] Step (4.12), splicing the first source domain image feature map, the second source domain image feature map, the third source domain image feature map and the fourth source domain image feature map to obtain source domain splicing features; splicing the first target domain image feature map, the second target domain image feature map, the third target domain image feature map and the fourth target domain image feature map to obtain target domain splicing features;

[0053] In step (4.13), convolution is used to transform the dimensions of the source domain splicing features and the target domain splicing features.

[0054] Preferably, step 5 comprises:

[0055] Step (5.1), calculate the cross entropy loss L for semantic segmentation of the source domain image S ;

[0056] L S =-∑y S log(f θ (x S )),

[0057] Among them, x S is the source domain image for cross-domain remote sensing image semantic segmentation, y S represents the one-hot encoding of the true label of the source domain image, f θ represents the student model;

[0058] Step (5.2), calculate the semantic segmentation cross entropy loss L of the mixed image mix ;

[0059]

[0060] Among them, x mix is a mixed domain image, One-hot encoding of mixed image labels;

[0061] Step (5.3), calculate the prototype contrast loss L cs ; Step (5.4), use cross entropy loss, semantic segmentation cross entropy loss and prototype contrast loss to train the cross-domain remote sensing image semantic segmentation model until the preset number of iterations of the cross-domain remote sensing image semantic segmentation model is reached, and a trained cross-domain remote sensing image semantic segmentation model is obtained.

[0062] Prioritize, during the training process of the cross-domain remote sensing image semantic segmentation model, random rotation, flipping and random photometric distortion are used on the cropped images.

[0063] Preferably, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the program.

[0064] Preferably, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0065] The beneficial effects achieved by the present invention are:

[0066] First, the existing technologies often ignore the image noise problem and perform independent adjustments on a single domain. The present invention effectively utilizes the information interaction between the source domain and the target domain through a hybrid strategy, which better helps the model learn the semantic features of the same category between different domains. In addition, the adaptive hybrid strategy can more effectively solve the problem of mixed image class imbalance, making the model training process more stable and reliable.

[0067] Second, compared with the existing technology, the prototype contrast learning method proposed in the present invention fully considers the uncertainty of the segmentation results, alleviates the problem of pseudo-label noise, ensures that the model is more flexible in aligning source domain features and target domain prototype features, and effectively helps the model learn more robust feature representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0069] Figure 1 It is a finishing flow chart of the present invention;

[0070] Figure 2 It is the overall structure diagram of the network of the present invention;

[0071] Figure 3 It is a specific flow chart of prototype contrast learning. DETAILED DESCRIPTION

[0072] See also Figure 1The present application discloses a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison, characterized in that a robust remote sensing semantic segmentation model is trained using a labeled cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam and an unlabeled cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen, and the model achieves good results on the target domain image ISPRS Vaihingen. The method comprises the following steps:

[0073] Step 1, obtaining a labeled cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam and an unlabeled cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen, and preprocessing the original images in the labeled cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam and the unlabeled cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen;

[0074] Step 2: Build a cross-domain remote sensing image semantic segmentation model based on Segformer and initialize the model parameters through common Segformer pre-training weights;

[0075] Step 3, firstly, the preprocessed cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam and the cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen are mixed by an adaptive mixing strategy to obtain a mixed image;

[0076] Step 4. Secondly, extract the semantic features of the source domain image ISPRS Potsdam and the target domain image ISPRS Vaihingen for cross-domain remote sensing image semantic segmentation, align the semantic features of the source domain and the target domain through prototype contrast learning, reduce the domain difference between the source domain and the target domain, and alleviate the adverse effects of pseudo-label noise on model performance.

[0077] Step 5. Finally, the cross-domain remote sensing image semantic segmentation model is trained using the mixed image, the cross-domain remote sensing image semantic segmentation source domain image and the confidence. A mapping relationship between the mixed image as input, the cross-domain remote sensing image semantic segmentation source domain image as input and the confidence as output is constructed to obtain the trained cross-domain remote sensing image semantic segmentation model. The trained model is tested on the target domain image ISPRS Vaihingen.

[0078] Step 1 includes:

[0079] Step (1.1), obtain the labeled cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam and the unlabeled target domain image ISPRS Vaihingen, the cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam and the cross-domain remote sensing image semantic segmentation source domain image target domain image ISPRS Vaihingen both include original images, ISPRS Potsdam contains 38 images with a size of 6000*6000 and a spatial resolution of 5 cm, and the images provide three band modes, namely IR-RG, RGB, and IR-R-GB; ISPRS Vaihingen contains 33 images with a size of 2000*2000 and a spatial resolution of 9 cm, and it has only one band mode IR-RG;

[0080] Step (1.2), crop the original image and cut it into images of size 512*512; use the cut images for cross-domain remote sensing image semantic segmentation model training, and use random rotation, flipping and random photometric distortion to perform data enhancement on the cut images during the training process.

[0081] Step 2: Build a cross-domain remote sensing image semantic segmentation model based on the semantic segmentation framework Segformer, including:

[0082] Step (2.1), build the semantic segmentation framework Segformer, which consists of an encoder and a decoder, called a student model, and is initialized with pre-trained weights;

[0083] In step (2.2), the student model is copied as the teacher model. The parameter initialization is consistent with the student model. The subsequent parameter update is updated through the student model using EMA. The parameter update formula of the teacher model is as follows:

[0084]

[0085] Among them, t represents the current iteration round, α represents the update weight, represents the parameters of the teacher model at the tth iteration, Represents the parameters of the student model at the tth iteration.

[0086] Step 3, mixing the source domain image ISPRS Potsdam and the target domain image ISPRS Vaihingen through an adaptive mixing strategy to obtain a mixed image, including:

[0087] Step (3.1), send the source domain image ISPRS Potsdam and the target domain image ISPRS Vaihingen into the student model to obtain the category prediction confidence Send the source domain image and the target domain image to the teacher model to obtain the corresponding category prediction confidence

[0088] Step (3.2), calculate the accuracy value of the category prediction confidence:

[0089]

[0091] in, Represents the accuracy of the source domain image category pixels, Represents the accuracy of target domain image category pixels;

[0092] Step (3.3), calculate the proportionality coefficient r1:

[0093]

[0094] Step (3.4), calculate the minimum number of category pixels n of the source domain image based on the label min and the maximum number of pixels of the source domain image n max , calculate the number of pixels in each category n c In the corresponding image, the proportional coefficient r2 is as follows:

[0095]

[0096] Step (3.5), multiply the proportional coefficient r1 and the proportional coefficient r2 to obtain the final source domain category pixel ratio r; compare r with 1, if r>1, then r=1; if r≤1, then take the original value of r;

[0097] Step (3.6), the accuracy of each source domain image category pixel Sort in ascending order, using the final source domain category pixel ratio r and the number of pixels in each category in the source domain n c Multiply them to get the number of source domain category pixels in the mixed image The accuracy of pixels from each source domain image category sorted in ascending order Select the corresponding quantity Create a mask matrix of the same size as the source domain image for cross-domain remote sensing image semantic segmentation The values ​​of the corresponding positions in the mask matrix of the selected pixels and the source domain images of the cross-domain remote sensing image semantic segmentation are set to 1, and the rest are 0;

[0098] Step (3.7), loop step (3.1) to step (3.6), and obtain the mixed mask matrix of each category of the source domain image for all cross-domain remote sensing image semantic segmentation The mask matrix corresponding to the target domain image is obtained by semantic segmentation of cross-domain remote sensing images Obtain the mask matrix corresponding to the target domain image of the cross-domain remote sensing image semantic segmentation; multiply the target domain image mask matrix and the source domain image mask matrix with the corresponding target domain image and source domain image, and then add them to obtain a mixed image; multiply the target domain image mask matrix and the source domain image mask matrix with the corresponding target domain pseudo label and source domain real label, and then add them to obtain the label of the mixed image.

[0099] Step 4: Align the semantic features of the source domain and the target domain through prototype comparison learning, including:

[0100] Step (4.1), the model encoder includes four stages, each stage will output a feature map, the cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam is sent to the student model to obtain the source domain image feature maps of the four stages, and the feature fusion of the source domain image feature maps of the four stages is performed to obtain the source domain feature map; the cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen is input to the teacher model to obtain the target domain image feature map of the four stages, and the feature fusion of the target domain image feature maps of the four stages is performed to obtain the target domain feature map;

[0101] Step (4.2), input the target domain image of the cross-domain remote sensing image semantic segmentation into the teacher model to obtain the prediction result, which is the pixel-level category confidence score, and extract the highest predicted category confidence score from the pixel-level category confidence scores and the second highest class confidence score

[0102] Step (4.3), calculate the reliability score of each category pixel Weaken the influence of low-confidence pixel features on prototype features and increase the influence of high-confidence pixel features in prototype features; the reliability score calculation formula is as follows:

[0103]

[0104] Step (4.4), based on the calculated reliability score Calculate the prototype features of the target domain. The specific calculation method is as follows:

[0105]

[0106] In the formula, is the pixel-level semantic feature of each category in the target domain, N c is the number of pixels for each category;

[0107] Step (4.5), after obtaining the prototype features of each category in the target domain, create a feature matrix with the same size as the source domain image of the cross-domain remote sensing image semantic segmentation according to the label of the source domain image of the cross-domain remote sensing image semantic segmentation, and use it to compare with the source domain pixel features;

[0108] In step (4.6), cosine similarity is used to compare the feature matrix and source domain pixel features of the same size as the source domain image of the cross-domain remote sensing image semantic segmentation to obtain the cosine similarity score. The specific calculation formula is as follows:

[0109]

[0110] in, is the source domain category pixel feature;

[0111] In step (4.7), the cosine similarity score is exponentially normalized; considering the pseudo-label noise problem, the entropy of the source domain pixels is used to calculate the similarity score L cs , which can better help the model learn common category features; the specific calculation process is as follows:

[0112]

[0113] Among them, k is the number of categories.

[0114] Step (4.1) includes:

[0115] Step (4.11), taking the width and height of the first source domain image feature map in the four stages as the standard, adjusting the width and height of the second source domain image feature map, the third source domain image feature map, and the fourth source domain image feature map to the same size as the first source domain image feature map; taking the width and height of the first target domain image feature map in the four stages as the standard, adjusting the width and height of the second target domain image feature map, the third target domain image feature map, and the fourth target domain image feature map to the same size as the first target domain image feature map;

[0116] Step (4.12), splicing the first source domain image feature map, the second source domain image feature map, the third source domain image feature map and the fourth source domain image feature map to obtain source domain splicing features; splicing the first target domain image feature map, the second target domain image feature map, the third target domain image feature map and the fourth target domain image feature map to obtain target domain splicing features;

[0117] In step (4.13), 1×1 convolution is used to transform the dimensions of the source domain splicing features and the target domain splicing features.

[0118] Step 5 includes:

[0119] Step (5.1), calculate the cross entropy loss L for semantic segmentation of the source domain image S ;

[0120] L S =-∑y S log(f θ (x S ));

[0121] Among them, x S is the source domain image, y S represents the one-hot encoding of the true label of the source domain image, f θ Represents the student model step (5.2), calculating the semantic segmentation cross entropy loss L of the mixed image mix ;

[0122]

[0123] Among them, x mix is a mixed domain image, One-hot encoding of mixed image labels

[0124] Step (5.3), calculate the prototype contrast loss L cs ;

[0125]

[0126] In step (5.4), the three calculated cross entropy losses, semantic segmentation cross entropy loss and prototype contrast loss are used to train the cross-domain remote sensing image semantic segmentation model until the preset number of iterations of the cross-domain remote sensing image semantic segmentation model is reached.

[0127] In an embodiment of the present application, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0128] In an embodiment of the present application, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0129] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0130] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention invented herein. This application is intended to cover any variations, uses or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art that are not invented by the present invention. The specification and examples are intended to be exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0131] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison, characterized in that: include: Input the pre-acquired target domain image into the trained cross-domain remote sensing image semantic segmentation model and output the predicted confidence. Among them, the cross-domain remote sensing image semantic segmentation model is trained, including: Step 1, obtaining a labeled cross-domain remote sensing image semantic segmentation source domain image and an unlabeled cross-domain remote sensing image semantic segmentation target domain image, and preprocessing the original images in the labeled cross-domain remote sensing image semantic segmentation source domain image and the unlabeled cross-domain remote sensing image semantic segmentation target domain image; Step 2: Build a cross-domain remote sensing image semantic segmentation model based on the semantic segmentation framework Segformer; Step 3, mixing the preprocessed cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image through an adaptive mixing strategy to obtain a mixed image; Step 4: extract semantic features of the source domain image and the target domain image of the cross-domain remote sensing image semantic segmentation, and align the semantic features of the source domain and the target domain through prototype comparison learning; Step 5, use the mixed image, cross-domain remote sensing image semantic segmentation source domain image and confidence to train the cross-domain remote sensing image semantic segmentation model, construct a mapping relationship between the mixed image as input, the cross-domain remote sensing image semantic segmentation source domain image as input and the confidence as output, and obtain a trained cross-domain remote sensing image semantic segmentation model.

2. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison according to claim 1, characterized in that: Step 1 includes: Step (1.1), obtaining a labeled cross-domain remote sensing image semantic segmentation source domain image and an unlabeled cross-domain remote sensing image semantic segmentation target domain image, wherein the cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image both include the original image; Step (1.2), cropping the original image, cutting the original image into images of a preset size.

3. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison according to claim 1 is characterized in that: Step 2: Build a cross-domain remote sensing image semantic segmentation model based on the semantic segmentation framework Segformer, including: Step (2.1), build the semantic segmentation framework Segformer, which consists of two parts: encoder and decoder, to obtain the student model; Step (2.2), obtain the teacher model based on the student model, and the parameter update formula of the teacher model is as follows: Among them, t represents the current iteration round, α represents the update weight, represents the parameters of the teacher model at the tth iteration, Represents the parameters of the student model at the tth iteration.

4. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison according to claim 3 is characterized in that: Step 3, mixing the preprocessed cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image through an adaptive mixing strategy to obtain a mixed image, including: Step (3.1), send the cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image into the student model to obtain the category prediction confidence The cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image are sent to the teacher model to obtain the corresponding category prediction confidence Step (3.2), calculate the accuracy value of the category prediction confidence: in, Represents the accuracy of the source domain image category pixels, Represents the accuracy of target domain image category pixels; Step (3.3), calculate the proportionality coefficient r1: Step (3.4), calculate the minimum number of category pixels n of the source domain image for cross-domain remote sensing image semantic segmentation based on the label min and the maximum number of category pixels n in the source domain image of cross-domain remote sensing image semantic segmentation max , calculate the number of pixels in each category n c The proportional coefficient r2 in the source domain image corresponding to the semantic segmentation of the cross-domain remote sensing image is as follows: Step (3.5), multiply the scale factor r1 by the scale factor r2 to obtain the final source domain category pixel ratio r; Compare the final source domain category pixel ratio r with 1. If the final source domain category pixel ratio r>1, then r=1; if the final source domain category pixel ratio r≤1, then the final source domain category pixel ratio r retains the original value; Step (3.6), the accuracy of all source domain image category pixels Sort in ascending order, using the final source domain category pixel ratio r and the number of pixels in each category in the source domain n c Multiply them to get the number of source domain category pixels in the mixed image The accuracy of pixels from each source domain image category sorted in ascending order Select the corresponding quantity Create a mask matrix of the same size as the source domain image for cross-domain remote sensing image semantic segmentation Mask Matrix The values ​​of the positions corresponding to the source domain images of the cross-domain remote sensing image semantic segmentation are set to 1, and the rest are 0; Step (3.7), loop step (3.1) to step (3.6), and obtain the mixed mask matrix of each category of the source domain image for all cross-domain remote sensing image semantic segmentation pass Obtain the mask matrix corresponding to the target domain image of the cross-domain remote sensing image semantic segmentation; multiply the target domain image mask matrix and the source domain image mask matrix with the corresponding target domain image and source domain image, and then add them to obtain a mixed image; multiply the target domain image mask matrix and the source domain image mask matrix with the corresponding target domain pseudo label and source domain real label, and then add them to obtain the label of the mixed image.

5. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison according to claim 1, characterized in that: Step 4: Align the semantic features of the source domain and the target domain through prototype comparison learning, including: Step (4.1), input the source domain image of the cross-domain remote sensing image semantic segmentation into the student model to obtain the source domain image feature maps of four stages, perform feature fusion on the source domain image feature maps of the four stages, and obtain the source domain feature map; input the target domain image of the cross-domain remote sensing image semantic segmentation into the teacher model to obtain the target domain image feature maps of four stages, perform feature fusion on the target domain image feature maps of the four stages, and obtain the target domain feature map; Step (4.2), input the target domain image of the cross-domain remote sensing image semantic segmentation into the teacher model to obtain the pixel-level category confidence score; extract the predicted highest category confidence score from the obtained pixel-level category confidence scores and the second highest class confidence score Step (4.3), calculate the reliability score of each category pixel Step (4.4), according to the reliability score Compute the prototype features of the target domain: In the formula, f c T is the pixel-level semantic feature of each category in the target domain, N c is the number of pixels for each category; Step (4.5) obtains the prototype feature f of each category in the target domain c T Finally, according to the label of the source domain image of the cross-domain remote sensing image semantic segmentation, a feature matrix with the same size as the source domain image of the cross-domain remote sensing image semantic segmentation is created; Step (4.6), use cosine similarity to compare the feature matrix and source domain pixel features of the same size as the source domain image of the cross-domain remote sensing image semantic segmentation, and obtain the cosine similarity score sim c : in, is the source domain category pixel feature; In step (4.7), the cosine similarity score is exponentially normalized; considering the pseudo-label noise problem, the entropy of the source domain pixels is used to calculate the prototype contrast loss L cs : Among them, k is the number of categories, N c is the number of pixels for each category, f c T is the pixel-level semantic feature of each category in the target domain, sim c is the cosine similarity score.

6. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison according to claim 5, characterized in that: Step (4.1) includes: Step (4.11), taking the width and height of the first source domain image feature map in the four stages as the standard, adjusting the width and height of the second source domain image feature map, the third source domain image feature map, and the fourth source domain image feature map to the same size as the first source domain image feature map; taking the width and height of the first target domain image feature map in the four stages as the standard, adjusting the width and height of the second target domain image feature map, the third target domain image feature map, and the fourth target domain image feature map to the same size as the first target domain image feature map; Step (4.12), splicing the first source domain image feature map, the second source domain image feature map, the third source domain image feature map and the fourth source domain image feature map to obtain source domain splicing features; splicing the first target domain image feature map, the second target domain image feature map, the third target domain image feature map and the fourth target domain image feature map to obtain target domain splicing features; In step (4.13), convolution is used to transform the dimensions of the source domain splicing features and the target domain splicing features.

7. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison according to claim 5, characterized in that: Step 5 includes: Step (5.1), calculate the cross entropy loss L for semantic segmentation of the source domain image S ; L S =-∑y S log(f θ (x S )), Among them, x S is the source domain image for cross-domain remote sensing image semantic segmentation, y S represents the one-hot encoding of the true label of the source domain image, f θ represents the student model; Step (5.2), calculate the semantic segmentation cross entropy loss L of the mixed image mix ; Among them, x mix is a mixed domain image, One-hot encoding representing mixed image labels; Step (5.3), calculate the prototype contrast loss L cs ; Step (5.4), use cross entropy loss, semantic segmentation cross entropy loss and prototype contrast loss to train the cross-domain remote sensing image semantic segmentation model until the preset number of iterations of the cross-domain remote sensing image semantic segmentation model is reached, and a trained cross-domain remote sensing image semantic segmentation model is obtained.

8. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison according to claim 1, characterized in that: During the training process of the cross-domain remote sensing image semantic segmentation model, random rotation, flipping and random photometric distortion are used on the cropped images.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images