Cross-domain remote sensing image semantic segmentation method based on image mixing and prototype contrast

Abstract

The application discloses a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison, inputs target domain images into a cross-domain remote sensing image semantic segmentation model, and outputs confidence; and the cross-domain remote sensing image semantic segmentation model is trained as follows: pre-processing source domain images with labels and target domain images without labels; mixing the pre-processed source domain images and target domain images through an adaptive mixing strategy to obtain mixed images; extracting semantic features of the source domain images and the target domain images, and aligning the semantic features of the source domain and the target domain through prototype comparison learning; and training a cross-domain remote sensing image semantic segmentation model based on a semantic segmentation framework Segformer by using the mixed images, the source domain images and the confidence. The application effectively utilizes information interaction between the source domain and the target domain through the mixing strategy, and better helps the model to learn semantic features of the same categories between different domains.

Description

TECHNICAL FIELD

[0001] The present application relates to a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison, and belongs to the technical field of image semantic segmentation. BACKGROUND

[0002] In recent years, the application range of remote sensing images has become more and more extensive, and semantic segmentation as a basic task has also received more and more attention. For example, building extraction, forest coverage area calculation, and road segmentation are all tasks with practical significance. Although deep learning has made great progress in computer vision tasks, semantic segmentation technology has also matured. However, the application of a model trained on source domain images to target domain images is significantly reduced, and cross-domain semantic segmentation has become a mainstream topic.

[0003] The cross-domain phenomenon in the field of remote sensing images is relatively obvious. Because images captured by different regions, different times, or even different sensors may have significant differences. At the same time, in order to train an effective semantic segmentation model, a large amount of labeled data is required. These data not only include the images themselves, but also the class labels of each pixel point. However, obtaining and labeling these data is usually costly and time-consuming. Therefore, a model trained on source domain images needs to be able to adapt to changes in image differences to maintain its generalization ability. The main problems faced by cross-domain remote sensing image semantic segmentation are as follows:

[0004] (1) The distribution difference of remote sensing images is very obvious, and the class imbalance is worse. Images are affected by various factors, resulting in more irrelevant noise in the images. For the same class, the features extracted by the model have significant differences, making it difficult to accurately identify the features between different domains.

[0005] (2) The pseudo-labels of self-training usually have noise, where the noise usually refers to incorrect labels. However, since the weights of the teacher model are iteratively updated by the student model, the noise problem of the pseudo-labels has a serious impact on the model, which can damage the stability and accuracy of the model training. SUMMARY

[0006] The technical problem to be solved by the present application is to overcome the defects of the prior art and provide a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison. A corresponding solution is proposed for the problem of obvious distribution difference and pseudo-label noise.

[0007] Preferably, the present application provides a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype comparison, comprising: inputting a pre-acquired target domain image into a trained cross-domain remote sensing image semantic segmentation model to output a predicted confidence level.

[0008] Wherein, training the cross-domain remote sensing image semantic segmentation model comprises:

[0009] Step 1, obtain cross-domain remote sensing image semantic segmentation source domain images with labels and cross-domain remote sensing image semantic segmentation target domain images without labels, and preprocess the original images in the cross-domain remote sensing image semantic segmentation source domain images with labels and the cross-domain remote sensing image semantic segmentation target domain images without labels;

[0010] Step 2, build a cross-domain remote sensing image semantic segmentation model based on a semantic segmentation framework Segformer;

[0011] Step 3, mix the preprocessed cross-domain remote sensing image semantic segmentation source domain images and cross-domain remote sensing image semantic segmentation target domain images through an adaptive mixing strategy to obtain mixed images;

[0012] Step 4, extract semantic features of the cross-domain remote sensing image semantic segmentation source domain images and the cross-domain remote sensing image semantic segmentation target domain images, and align the semantic features of the source domain and the target domain through prototype contrast learning;

[0013] Step 5, train the cross-domain remote sensing image semantic segmentation model using the mixed images, the cross-domain remote sensing image semantic segmentation source domain images, and the confidence, build a mapping relationship between the mixed images as input, the cross-domain remote sensing image semantic segmentation source domain images as input, and the confidence as output, and obtain the trained cross-domain remote sensing image semantic segmentation model.

[0014] Preferably, step 1 comprises:

[0015] Step (1.1), obtain cross-domain remote sensing image semantic segmentation source domain images with labels and cross-domain remote sensing image semantic segmentation target domain images without labels, and the cross-domain remote sensing image semantic segmentation source domain images and the cross-domain remote sensing image semantic segmentation target domain images both include original images;

[0016] Step (1.2), crop the original images to cut the original images into images of a preset size.

[0017] Preferably, step 2, building a cross-domain remote sensing image semantic segmentation model based on a semantic segmentation framework Segformer, comprises:

[0018] Step (2.1), build a semantic segmentation framework Segformer, the semantic segmentation framework Segformer includes an encoder and a decoder, and obtain a student model;

[0019] Step (2.2), obtain a teacher model based on the student model, and the parameter update formula of the teacher model is as follows:

[0020]

[0021] wherein t represents the current iteration round, a represents the update weight, represents the parameters of the teacher model in the tth iteration, represents the parameters of the student model in the tth iteration.

[0022] Preferably, step 3, the pre-processed cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image are mixed by an adaptive mixing strategy to obtain a mixed image, comprising:

[0023] Step (3.1), the cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image are sent into the student model to obtain the class prediction confidence The cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image are sent into the teacher model to obtain the corresponding class prediction confidence

[0024] Step (3.2), the accuracy value of the class prediction confidence is calculated:

[0025]

[0026] wherein, represents the accuracy of the source domain image class pixels, represents the accuracy of the target domain image class pixels;

[0027] Step (3.3), the proportion coefficient r1 is calculated:

[0028]

[0029] Step (3.4), the minimum class pixel number n min and the maximum class pixel number n max of the cross-domain remote sensing image semantic segmentation source domain image are calculated according to the label, and the class pixel number n c of each class is calculated.

[0030]

[0031] Step (3.5), the proportion coefficient r1 is multiplied by the proportion coefficient r2 to obtain the final source domain class pixel proportion r.

[0032] The final source domain class pixel proportion r is compared with 1, if the final source domain class pixel proportion r>1, then r=1; if the final source domain class pixel proportion r≤1, then the final source domain class pixel proportion r keeps the original value.

[0033] Step (3.6) assesses the accuracy of pixels for all source domain image categories. Sort in ascending order, using the final source domain category pixel ratio r and the number of pixels n for each category in the source domain. c Multiplying yields the number of source domain category pixels in the blended image. Accuracy of pixels for each source domain image category sorted in ascending order Select the corresponding quantity Create a mask matrix of the same size as the source domain image for cross-domain remote sensing image semantic segmentation. Mask matrix In the semantic segmentation of cross-domain remote sensing images, the corresponding position of the source domain image is set to 1, and the rest are 0;

[0034] Step (3.7) repeats steps (3.1) to (3.6) to obtain the mixing mask matrix for each category of the source domain image for semantic segmentation of all cross-domain remote sensing images. The mask matrix corresponding to the target domain image is obtained by semantic segmentation of cross-domain remote sensing images. Obtain the mask matrix corresponding to the target domain image in the cross-domain remote sensing image semantic segmentation; multiply the target domain image mask matrix and the source domain image mask matrix with the corresponding target domain image and the source domain image, and then add them to obtain the hybrid image; multiply the target domain image mask matrix and the source domain image mask matrix with the corresponding target domain pseudo label and the source domain real label, and then add them to obtain the label of the hybrid image.

[0035] Prioritizing step 4, aligning the semantic features of the source and target domains through prototype contrastive learning includes:

[0036] Step (4.1) involves inputting the source domain image of cross-domain remote sensing image semantic segmentation into the student model to obtain source domain image feature maps in four stages, performing feature fusion on the source domain image feature maps in four stages to obtain source domain feature maps; inputting the target domain image of cross-domain remote sensing image semantic segmentation into the teacher model to obtain target domain image feature maps in four stages, performing feature fusion on the target domain image feature maps in four stages to obtain target domain feature maps.

[0037] Step (4.2) involves inputting the target domain image of the cross-domain remote sensing image semantic segmentation into the teacher model to obtain pixel-level class confidence scores; and extracting the highest predicted class confidence score from the obtained pixel-level class confidence scores. and the second highest category confidence score

[0038] Step (4.3) calculates the reliability score for each category of pixel.

[0039]

[0040] Step (4.4), according to the reliability score Calculate the prototype feature of the target domain:

[0041]

[0042] In the formula, is the pixel-level semantic feature of each class of the target domain, N c is the number of pixels of each class;

[0043] Step (4.5), get the prototype feature of each class of the target domain After that, according to the label of the source domain image of the cross-domain remote sensing image semantic segmentation, a feature matrix consistent with the size of the source domain image of the cross-domain remote sensing image semantic segmentation is created;

[0044] Step (4.6), compare the feature matrix consistent with the size of the source domain image of the cross-domain remote sensing image semantic segmentation with the source domain pixel feature using cosine similarity, and get the cosine similarity score sim c :

[0045]

[0046] is the source domain class pixel feature;

[0047] Step (4.7), exponential normalization of the cosine similarity score; considering the problem of pseudo-label noise, the entropy of the source domain pixel is used to calculate the prototype comparison loss L cs :

[0048]

[0049] wherein, k is the number of classes, N c is the number of pixels of each class, is the pixel-level semantic feature of each class of the target domain, sim c is the cosine similarity score.

[0050] Preferably, in step (4.1), it includes:

[0051] Step (4.11), taking the width and height of the first source domain image feature map in the four-stage source domain image feature map as the standard, adjusting the width and height of the second source domain image feature map, the third source domain image feature map and the fourth source domain image feature map to the same size as the first source domain image feature map; taking the width and height of the first target domain image feature map in the four-stage target domain image feature map as the standard, adjusting the width and height of the second target domain image feature map, the third target domain image feature map and the fourth target domain image feature map to the same size as the first target domain image feature map.

[0052] Step (4.12), splicing the first source domain image feature map, the second source domain image feature map, the third source domain image feature map and the fourth source domain image feature map to obtain a source domain splicing feature; splicing the first target domain image feature map, the second target domain image feature map, the third target domain image feature map and the fourth target domain image feature map to obtain a target domain splicing feature;

[0053] Step (4.13), performing dimension transformation on the source domain splicing feature and the target domain splicing feature by convolution.

[0054] Preferably, step 5 comprises:

[0055] Step (5.1), calculating a cross-entropy loss L S of the source domain image semantic segmentation.

[0056] L S = -∑y S log(f θ (x S )),

[0057] wherein x S is the source domain image of the cross-domain remote sensing image semantic segmentation, y S represents one-hot encoding of the real label of the source domain image, and f θ represents the student model.

[0058] Step (5.2), calculating a semantic segmentation cross-entropy loss L mix of the mixed image.

[0059]

[0060] wherein x mix is the mixed domain image, represents one-hot encoding of the label of the mixed image.

[0061] Step (5.3), calculating a prototype contrast loss L cs Step (5.4), training the cross-domain remote sensing image semantic segmentation model by using the cross-entropy loss, the semantic segmentation cross-entropy loss and the prototype contrast loss until the preset iteration number of the cross-domain remote sensing image semantic segmentation model reaches, and obtaining the trained cross-domain remote sensing image semantic segmentation model.

[0062] Preferably, during the training of the cross-domain remote sensing image semantic segmentation model, the cut images are subjected to random rotation, flipping and random photometric distortion.

[0063] Preferably, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method of any of the first aspect when executing the program.

[0064] Preferably, the present application provides a computer readable storage medium having stored thereon a computer program, the computer program implementing the steps of the method of any of the first aspect when executed by a processor.

[0065] The present application achieves the following beneficial effects:

[0066] First, the prior art often ignores the image noise problem, and all are independent adjustment on a single domain. The present application effectively utilizes the information interaction between the source domain and the target domain through the hybrid strategy, better helps the model to learn the semantic features of the same category between different domains. And the adaptive hybrid strategy can more effectively solve the mixed image class imbalance problem, so that the training process of the model is more stable and reliable.

[0067] Second, compared with the prior art, the prototype contrast learning method proposed in the present application fully considers the uncertainty of the segmentation result, alleviates the pseudo label noise problem, ensures that the model is more flexible in aligning the source domain features and the target domain prototype features, and effectively helps the model to learn more robust feature representation. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0069] Figure 1 is the flowchart of the present application;

[0070] Figure 2 is the overall structure diagram of the network of the present application;

[0071] Figure 3 is the specific flowchart of the prototype contrast learning. DETAILED DESCRIPTION

[0072] Reference Figure 1The application discloses a cross-domain remote sensing image semantic segmentation method based on image mixing and prototype contrast, and has the characteristics that a robust remote sensing semantic segmentation model is trained by using a cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam with labels and a cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen without labels, the model achieves good results on the target domain image ISPRS Vaihingen, and the method comprises the following steps:

[0073] Step 1, a cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam with labels and a cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen without labels are obtained, and the original images in the cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam with labels and the cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen without labels are preprocessed;

[0074] Step 2, a cross-domain remote sensing image semantic segmentation model based on Segformer is built, and the model parameters are initialized by using common Segformer pre-training weights;

[0075] Step 3, the preprocessed cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam and the cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen are mixed by using a self-adaptive mixing strategy to obtain a mixed image;

[0076] Step 4, secondly, semantic features of the cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam and the cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen are extracted, the semantic features of the source domain and the target domain are aligned by using prototype contrast learning, the domain difference between the source domain and the target domain is reduced, and the adverse effects of pseudo-label noise on model performance are alleviated;

[0077] Step 5, finally, the cross-domain remote sensing image semantic segmentation model is trained by using the mixed image, the cross-domain remote sensing image semantic segmentation source domain image and the confidence, a mapping relationship between the mixed image as input, the cross-domain remote sensing image semantic segmentation source domain image as input and the confidence as output is constructed, the trained cross-domain remote sensing image semantic segmentation model is obtained, and the trained model is detected on the target domain image ISPRS Vaihingen.

[0078] Step 1 comprises:

[0079] Step (1.1), obtain the cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam with labels and the target domain image ISPRS Vaihingen without labels, the cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam and the cross-domain remote sensing image semantic segmentation source domain image target domain image ISPRS Vaihingen both include original images, ISPRS Potsdam contains 38 images with a size of 6000*6000 and a spatial resolution of 5 cm, the images provide three band modes, which are IR-R-G, R-G-B, and IR-R-GB; ISPRS Vaihingen contains 33 images with a size of 2000*2000 and a spatial resolution of 9 cm, and it only has one band mode IR-R-G;

[0080] Step (1.2), crop the original image, cut the original image into images with a size of 512*512; use the cut images for cross-domain remote sensing image semantic segmentation model training, and perform data enhancement on the cut images by using random rotation, flipping and random photometric distortion during the training process.

[0081] Step 2, build a cross-domain remote sensing image semantic segmentation model based on the semantic segmentation framework Segformer, including:

[0082] Step (2.1), build the semantic segmentation framework Segformer, which contains an encoder and a decoder, referred to as a student model, and is initialized with pre-training weights;

[0083] Step (2.2), copy the student model to the teacher model, the parameter initialization is consistent with the student model, and the subsequent parameter update is updated by the student model using EMA, and the parameter update formula of the teacher model is as follows:

[0084]

[0085] Where t represents the current iteration round, and a represents the update weight, represents the parameters of the teacher model in the tth iteration, represents the parameters of the student model in the tth iteration.

[0086] Step 3, mix the source domain image ISPRS Potsdam and the target domain image ISPRS Vaihingen through an adaptive mixing strategy to obtain mixed images, including:

[0087] Step (3.1), send the source domain image ISPRS Potsdam and the target domain image ISPRS Vaihingen into the student model to obtain class prediction confidence The source domain image and the target domain image are fed into the teacher model to obtain the corresponding category prediction confidence scores.

[0088] Step (3.2), calculate the accuracy value of the category prediction confidence score:

[0089]

[0090] in, The accuracy of pixels representing the source domain image category. The accuracy of pixels representing the target domain image category;

[0091] Step (3.3), calculate the proportionality coefficient r1:

[0092]

[0093] Step (3.4): Calculate the minimum number of class pixels n in the source domain image based on the label. min The maximum number of pixels n in the source domain image max Calculate the number of pixels n for each category. c The scaling factor r2 in the corresponding image is calculated using the following formula:

[0094]

[0095] Step (3.5): Multiply the scaling factor r1 and the scaling factor r2 to obtain the final source domain category pixel ratio r; compare r with 1. If r > 1, then r = 1; if r ≤ 1, then take the original value of r.

[0096] Step (3.6) assesses the accuracy of pixels for each source domain image category. Sort in ascending order, using the final source domain category pixel ratio r and the number of pixels n for each category in the source domain. c Multiplying yields the number of source domain category pixels in the blended image. Accuracy of pixels for each source domain image category sorted in ascending order Select the corresponding quantity Create a mask matrix of the same size as the source domain image for cross-domain remote sensing image semantic segmentation. In the mask matrix of the selected pixels, the values ​​at the corresponding positions in the source domain image of the cross-domain remote sensing image semantic segmentation are set to 1, and the rest are 0;

[0097] Step (3.7) repeats steps (3.1) to (3.6) to obtain the mixing mask matrix for each category of the source domain image for semantic segmentation of all cross-domain remote sensing images. The mask matrix corresponding to the target domain image is obtained by semantic segmentation of cross-domain remote sensing images. A mask matrix corresponding to a target domain image of the cross-domain remote sensing image semantic segmentation is obtained; a mixed image is obtained by multiplying and then adding the target domain image mask matrix, the source domain image mask matrix, and the corresponding target domain image and source domain image; and a label of the mixed image is obtained by multiplying and then adding the target domain image mask matrix, the source domain image mask matrix, and the corresponding target domain pseudo label and source domain real label.

[0098] Step 4, aligning the semantic features of the source domain and the target domain through prototype contrast learning, comprising:

[0099] Step (4.1), the model encoder comprises four stages, each stage outputs a feature map, the cross-domain remote sensing image semantic segmentation source domain image ISPRS Potsdam is input into a student model to obtain four-stage source domain image feature maps, the four-stage source domain image feature maps are fused to obtain a source domain feature map; the cross-domain remote sensing image semantic segmentation target domain image ISPRS Vaihingen is input into a teacher model to obtain four-stage target domain image feature maps, and the four-stage target domain image feature maps are fused to obtain a target domain feature map;

[0100] Step (4.2), inputting the cross-domain remote sensing image semantic segmentation target domain image into the teacher model to obtain a prediction result, the prediction result is a pixel-level class confidence score, and the highest class confidence score and the second highest class confidence score

[0101] Step (4.3), calculating a reliability score of each class pixel Weakening the influence of low-confidence pixel features on prototype features and improving the influence of high-confidence pixel features on prototype features; the reliability score calculation formula is as follows:

[0102]

[0103] Step (4.4), calculating the prototype feature of the target domain according to the calculated reliability score The specific calculation method is as follows:

[0104]

[0105] In the formula, is the pixel-level semantic feature of each class of the target domain, N c is the number of pixels of each class;

[0106] Step (4.5), after obtaining the prototype features of each category of the target domain, the label of the source domain image of the cross-domain remote sensing image semantic segmentation is used to create a feature matrix consistent with the size of the source domain image of the cross-domain remote sensing image semantic segmentation, which is used for comparison with the source domain pixel features;

[0107] Step (4.6), the cosine similarity is used to compare the feature matrix consistent with the size of the source domain image of the cross-domain remote sensing image semantic segmentation and the source domain pixel features, and the cosine similarity score is obtained, and the specific calculation formula is as follows:

[0108]

[0109] Wherein, is the source domain category pixel feature;

[0110] Step (4.7), the cosine similarity score is exponentially normalized; considering the pseudo label noise problem, the entropy of the source domain pixel is used to calculate the similarity score L cs , which can better help the model learn the common category features; the specific calculation process is as follows:

[0111]

[0112] Wherein, k is the number of categories.

[0113] In step (4.1), it includes:

[0114] Step (4.11), taking the width and height of the first source domain image feature map in the four-stage source domain image feature map as the standard, the width and height of the second source domain image feature map, the third source domain image feature map and the fourth source domain image feature map are adjusted to the same size as the first source domain image feature map; taking the width and height of the first target domain image feature map in the four-stage target domain image feature map as the standard, the width and height of the second target domain image feature map, the third target domain image feature map and the fourth target domain image feature map are adjusted to the same size as the first target domain image feature map;

[0115] Step (4.12), the first source domain image feature map, the second source domain image feature map, the third source domain image feature map and the fourth source domain image feature map are spliced to obtain the source domain splicing feature; the first target domain image feature map, the second target domain image feature map, the third target domain image feature map and the fourth target domain image feature map are spliced to obtain the target domain splicing feature;

[0116] Step (4.13), the source domain splicing feature and the target domain splicing feature are dimensionally transformed by 1x1 convolution.

[0117] Step 5 includes:

[0118] Step (5.1), calculating the cross-entropy loss L of semantic segmentation of the source domain image S ;

[0119] L S = -∑y S log(f θ (x S ));

[0120] wherein x S is the source domain image, y S represents the one-hot encoding of the real label of the source domain image, and f θ represents the student model Step (5.2), calculating the cross-entropy loss L of semantic segmentation of the mixed image mix ;

[0121]

[0122] wherein x mix is the mixed domain image, represents the one-hot encoding of the label of the mixed image

[0123] Step (5.3), calculating the prototype contrast loss L cs ;

[0124]

[0125] Step (5.4), training the cross-domain remote sensing image semantic segmentation model with the three calculated cross-entropy losses, the cross-entropy loss of semantic segmentation and the prototype contrast loss until the preset iteration number of the cross-domain remote sensing image semantic segmentation model is reached.

[0126] In the embodiments of the present application, the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of any one of the above.

[0127] In the embodiments of the present application, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the method of any one of the above.

[0128] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between each of the embodiments can be referred to each other, and each of the embodiments mainly describes the difference from other embodiments.

[0129] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0130] The above detailed description has shown, by way of example, various features of the application. It is intended that the claim be construed in a manner consistent with the scope and spirit of the application as set forth in the appended claims.

Claims

1. A cross-domain remote sensing image semantic segmentation method based on image mixing and prototype contrast, characterized in that, The method comprises the following steps: inputting the pre-acquired target domain image into the trained cross-domain remote sensing image semantic segmentation model to output the predicted confidence; wherein the training of the cross-domain remote sensing image semantic segmentation model comprises: Step 1: acquiring a cross-domain remote sensing image semantic segmentation source domain image with labels and a cross-domain remote sensing image semantic segmentation target domain image without labels, and pre-processing the original images in the cross-domain remote sensing image semantic segmentation source domain image with labels and the cross-domain remote sensing image semantic segmentation target domain image without labels; Step 2: building a cross-domain remote sensing image semantic segmentation model based on a semantic segmentation framework Segformer, comprising: Step (2.1), building a semantic segmentation framework Segformer, which comprises an encoder and a decoder, to obtain a student model; Step (2.2), obtaining a teacher model based on the student model, and the parameter updating formula of the teacher model is as follows: ， wherein, represents the current iteration round, represents the updated weights, represents the parameters of the teacher model at the tth iteration, represents the parameters of the student model at the tth iteration; Step 3: mixing the pre-processed cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image through an adaptive mixing strategy to obtain a mixed image, comprising: Step (3.1), input the cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image into the student model to obtain a class prediction confidence ; input the cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image into the teacher model to obtain a corresponding class prediction confidence ; Step (3.2), calculating the accuracy value of the class prediction confidence: ， ， wherein, accuracy of representing pixels of a source domain image class, accuracy of representing pixels of a target domain image class; Step (3.3), calculating the scaling factor : ； Step (3.4), calculating the least class pixel number of the cross-domain remote sensing image semantic segmentation source domain image according to the label and the most class pixel number of the cross-domain remote sensing image semantic segmentation source domain image , calculating the class pixel number of each class the proportion coefficient in the corresponding cross-domain remote sensing image semantic segmentation source domain image , the formula is as follows: (4) Step (3.5), multiplying the proportionality coefficient with the proportionality coefficient to obtain the final source domain class pixel proportion ; the final source domain class pixel proportion is compared with 1, if the final source domain class pixel proportion is greater than 1, then the final source domain class pixel proportion is less than 1, then the final source domain class pixel proportion is retained as the original value; Step (3.6) assesses the accuracy of pixels for all source domain image categories. Sort in ascending order, using the final source domain category pixel ratio. Number of pixels per category in the source domain Multiplying yields the number of source domain category pixels in the blended image. Accuracy of pixels for each source domain image category sorted in ascending order. Select the corresponding quantity Create a mask matrix of the same size as the source domain image for cross-domain remote sensing image semantic segmentation. mask matrix In the semantic segmentation of cross-domain remote sensing images, the corresponding position of the source domain image is set to 1, and the rest are 0; Step (3.7), cycle steps (3.1) to (3.6) to obtain a mixed mask matrix of each class of the source domain image of the cross-domain remote sensing image semantic segmentation , through the cross-domain remote sensing image semantic segmentation target domain image corresponding mask matrix , obtain the cross-domain remote sensing image semantic segmentation target domain image corresponding mask matrix; multiply and add the target domain image mask matrix, the source domain image mask matrix and the corresponding target domain image, the source domain image to obtain a mixed image; multiply and add the target domain image mask matrix, the source domain image mask matrix and the corresponding target domain pseudo label, the source domain real label to obtain the label of the mixed image; Step 4: extracting the semantic features of the cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image, and aligning the semantic features of the source domain and the target domain through prototype contrast learning; Step 5: training the cross-domain remote sensing image semantic segmentation model using the mixed image, the cross-domain remote sensing image semantic segmentation source domain image and the confidence, building a mapping relationship between the mixed image as input, the cross-domain remote sensing image semantic segmentation source domain image as input and the confidence as output, and obtaining the trained cross-domain remote sensing image semantic segmentation model.

2. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype contrast according to claim 1, characterized in that, Step 1 comprises: Step (1.1), acquiring a cross-domain remote sensing image semantic segmentation source domain image with labels and a cross-domain remote sensing image semantic segmentation target domain image without labels, wherein the cross-domain remote sensing image semantic segmentation source domain image and the cross-domain remote sensing image semantic segmentation target domain image both comprise original images; Step (1.2), cropping the original images to cut the original images into images with a preset size.

3. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype contrast according to claim 1, characterized in that, Step 4: aligning the semantic features of the source domain and the target domain through prototype contrast learning, comprising: Step (4.1), inputting the cross-domain remote sensing image semantic segmentation source domain image into the student model to obtain four-stage source domain image feature maps, performing feature fusion on the four-stage source domain image feature maps to obtain a source domain feature map; inputting the cross-domain remote sensing image semantic segmentation target domain image into the teacher model to obtain four-stage target domain image feature maps, and performing feature fusion on the four-stage target domain image feature maps to obtain a target domain feature map; Step (4.2), input the cross-domain remote sensing image semantic segmentation target domain image into the teacher model to obtain a pixel-level class confidence score; extract a predicted highest class confidence score in the pixel-level class confidence score and a second highest class confidence score ; Step (4.3), the reliability score of each class of pixels is calculated : ， Step (4.4), computing prototype features of target domain according to reliability score , computing prototype features of target domain according to reliability score ， In the formula, is the pixel-level semantic feature of each class of the target domain, N c is the number of pixels of each class. Step (4.5), obtaining the prototype features of each category of the target domain Afterwards, according to the label of the source domain image of the cross-domain remote sensing image semantic segmentation, a feature matrix consistent with the size of the source domain image of the cross-domain remote sensing image semantic segmentation is created. Step (4.6), using cosine similarity to compare the feature matrix consistent with the size of the source domain image of cross-domain remote sensing image semantic segmentation and the source domain pixel feature, and get the cosine similarity score sim c : ， wherein, is the source domain class pixel feature; Step (4.7), exponential normalization is performed on the cosine similarity score; considering the problem of pseudo label noise, the prototype contrast loss L is calculated by using the entropy of the source domain pixels cs : ， where k is the number of classes, n c is the number of pixels for each class, is the source domain class pixel feature, sim c is the cosine similarity score.

4. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype contrast according to claim 3, characterized in that, In Step (4.1), comprising: Step (4.11), adjusting the width and height of the second source domain image feature map, the third source domain image feature map and the fourth source domain image feature map to the same size as the first source domain image feature map according to the width and height of the first source domain image feature map in the first stage of the four stages of source domain image feature maps; adjusting the width and height of the second target domain image feature map, the third target domain image feature map and the fourth target domain image feature map to the same size as the first target domain image feature map according to the width and height of the first target domain image feature map in the first stage of the four stages of target domain image feature maps; Step (4.12), splicing the first source domain image feature map, the second source domain image feature map, the third source domain image feature map and the fourth source domain image feature map to obtain a source domain spliced feature; splicing the first target domain image feature map, the second target domain image feature map, the third target domain image feature map and the fourth target domain image feature map to obtain a target domain spliced feature; Step (4.13), performing dimension transformation on the source domain spliced feature and the target domain spliced feature by convolution.

5. The cross-domain remote sensing image semantic segmentation method based on image mixing and prototype contrast according to claim 3, characterized in that, Step 5, comprising: Step (5.1), calculating cross-entropy loss of semantic segmentation of source domain image ; ， wherein, is a source domain image for cross-domain remote sensing image semantic segmentation, denotes one-hot encoding of the real label of the source domain image, denotes a student model; Step (5.2), calculating the semantic segmentation cross-entropy loss of the mixed image ; ， wherein, is a mixed domain image, denotes a one-hot encoding of the mixed image label; Step (5.3), computing the prototype contrast loss ; Step (5.4), using The cross-domain remote sensing image semantic segmentation model is trained until a preset iteration number of the cross-domain remote sensing image semantic segmentation model reaches, and a trained cross-domain remote sensing image semantic segmentation model is obtained.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1 to 5 when executing the program.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 5.

Images