Remote sensing image semantic segmentation method and device based on domain adaptation

Abstract

The present invention discloses a method and device for semantic segmentation of remote sensing images based on domain adaptation. The method comprises acquiring a remote sensing dataset comprising labeled source domain data and unlabeled target domain data; constructing an average teacher framework with an auxiliary prototype classifier, wherein the average teacher framework with the auxiliary prototype classifier comprises a teacher model and a student model; using the remote sensing dataset to train and optimize parameters of the student model, wherein the teacher model updates its parameters using an exponential moving average, and the auxiliary prototype classifier updates its weight using an exponential moving average; inputting the unlabeled target domain data into the trained student model for point-by-point prediction to obtain a segmentation result corresponding to the unlabeled target domain data; thereby, by constructing the average teacher framework with the auxiliary prototype classifier, the domain difference problem in the cross-domain semantic segmentation task of remote sensing images can be solved, and the class hierarchy between the source domain and the target domain can be aligned, thereby improving the segmentation performance.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation technology, and in particular to a remote sensing image semantic segmentation method based on domain adaptation, a computer-readable storage medium, a computer device, and a remote sensing image semantic segmentation device based on domain adaptation. Background Art

[0002] Remote sensing images are widely used in various applications such as land cover mapping, urban planning, and environmental monitoring. Semantic segmentation is a fundamental task in remote sensing image analysis, where each pixel in an image is assigned a class label representing the object or region to which it belongs. Currently, fully supervised deep convolutional neural network methods have achieved remarkable results in the task of semantic segmentation of remote sensing images. However, when a model trained on a large-scale labeled dataset (source domain) is used to segment images collected from different scenes (target domain), its performance usually degrades due to inter-domain differences. The factors that contribute to domain differences include imaging conditions, geographic location, sensor specifications, and other aspects. Unsupervised Domain Adaptation (UDA) technology can alleviate these differences. The technology aims to learn a domain-invariant feature representation by adjusting the feature distributions of the source and target domains, so as to extract regions of interest from target domain images without semantic annotation of target domain data.

[0003] The techniques of UDA semantic segmentation mainly include adversarial learning, self-training and multi-stage methods; UDA methods based on adversarial learning can effectively align global marginal distributions, but they are prone to negative transfer (i.e., aligned classes are incorrectly aligned), especially in the case of class imbalance; to solve this problem, CLAN adopts a collaborative training method to implicitly determine the degree of alignment of each class; but the similarity between features of different categories can easily cause misalignment; UDA methods based on self-training explicitly perform class feature alignment by utilizing the pseudo-labels of the target domain and the class prototypes of the source domain; the pseudo-label of the target sample is estimated by calculating the distance between its features and the class prototype of the source domain; however, the class prototype of the source domain may not effectively reflect the feature center point of each semantic category in the target domain; this may reduce the reliability of the pseudo-label of the target sample and affect the performance of the classifier; when there is a large gap between the source domain and the target domain category distribution, the pseudo-label of the target sample is estimated by calculating the distance between its features and the class prototype of the source domain; however, the class prototype of the source domain may not effectively reflect the feature center point of each semantic category in the target domain; this may reduce the reliability of the pseudo-label of the target sample and affect the performance of the classifier; This problem is particularly evident in the unsupervised domain adaptation semantic segmentation task of remote sensing images when there is inconsistency between them; for example, in remote sensing images of urban scenes, the proportion of buildings and road targets is higher than that of rural scenes; multi-stage UDA methods generally include adversarial learning, self-training and knowledge distillation; in the first stage, they adopt the adversarial UDA method to train the model and use its weights as the initialization parameters of the next stage; in the second stage, the pre-trained model is used to calculate the pseudo-labels of the target domain and the prototypes of specific categories; in addition, during the training process, the prototypes are used to help correct the pseudo-labels online; in the final stage, the knowledge distillation technique is used to transfer the knowledge learned from the domain adaptation segmentation model trained in the previous stage to the self-supervised pre-trained model; although multi-stage training methods may produce better results than end-to-end methods, they heavily rely on the performance of the adversarial initialization method and require complex training strategies. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems in the above-mentioned technologies to a certain extent. To this end, one object of the present invention is to propose a domain-adaptive remote sensing image semantic segmentation method. By constructing an average teacher framework with an auxiliary prototype classifier, the domain difference problem in the cross-domain semantic segmentation task of remote sensing images can be solved. The framework can effectively extract the region of interest from the target domain data and achieve class hierarchy alignment between the source domain and the target domain, thereby improving the segmentation performance.

[0005] A second object of the present invention is to provide a computer-readable storage medium.

[0006] A third object of the present invention is to provide a computer device.

[0007] The fourth object of the present invention is to propose a remote sensing image semantic segmentation device based on domain adaptation.

[0008] To achieve the above-mentioned objectives, the first embodiment of the present invention proposes a domain-adaptive-based remote sensing image semantic segmentation method, comprising the following steps: obtaining a remote sensing dataset, wherein the remote sensing dataset includes labeled source domain data and unlabeled target domain data; constructing an average teacher framework with an auxiliary prototype classifier, wherein the average teacher framework with the auxiliary prototype classifier includes a teacher model and a student model; using the remote sensing dataset to train and optimize the parameters of the student model, wherein the teacher model uses an exponential moving average to update its parameters, and the auxiliary prototype classifier uses an exponential moving average to update its weights; inputting the unlabeled target domain data into the trained student model for point-by-point prediction to obtain a segmentation result corresponding to the unlabeled target domain data.

[0009] According to the domain adaptation-based semantic segmentation method for remote sensing images of an embodiment of the present invention, first, a remote sensing dataset is obtained, wherein the remote sensing dataset includes labeled source domain data and unlabeled target domain data; then, an average teacher framework with an auxiliary prototype classifier is constructed, wherein the average teacher framework with the auxiliary prototype classifier includes a teacher model and a student model; then, the remote sensing dataset is used to train and optimize the parameters of the student model, wherein the teacher model uses exponential moving average to update its parameters, and the auxiliary prototype classifier uses exponential moving average to update its weights; finally, the unlabeled target domain data is input into the trained student model for point-by-point prediction to obtain the segmentation result corresponding to the unlabeled target domain data; thus, by constructing the average teacher framework with the auxiliary prototype classifier, the domain difference problem in the cross-domain semantic segmentation task of remote sensing images can be solved, the region of interest can be effectively extracted from the target domain data, and the class hierarchy between the source domain and the target domain can be aligned, thereby improving the segmentation performance.

[0010] In addition, the remote sensing image semantic segmentation method based on domain adaptation proposed in the above embodiment of the present invention may also have the following additional technical features:

[0011] Optionally, constructing an average teacher framework with an auxiliary prototype classifier includes: the student model includes a feature encoder and a parameterized classifier, and DeepLabV2 is used as the network structure of the student model, and ResNet-101 is used as its skeleton; the network structure and skeleton of the teacher model are consistent with the student model; corresponding categories of memory libraries are constructed for the source domain data and the target domain data in the form of queues, so that after the features output by the corresponding feature encoder are embedded and filtered, they are saved in the memory libraries of the corresponding categories as feature vectors of different categories; the memory libraries of the corresponding categories of the source domain and the target domain are spliced, and the prototype corresponding to the category is clustered using the KMeans clustering algorithm to serve as an auxiliary prototype classifier.

[0012] Optionally, the remote sensing data set is used to train and optimize the parameters of the student model, including: in the first epoch of training, the student model is trained and the parameters are optimized using the labeled source domain data, and the feature vectors of different classes of the source domain data are saved in the source domain memory bank; after the first epoch of training, the teacher model uses the parameters of the student model to initialize the parameters, and uses the KMeans algorithm to cluster the source domain memory bank to obtain the initialized prototype of each class as an auxiliary prototype classifier; in the second epoch of training, the student model with the auxiliary prototype classifier is trained using the labeled source domain data; the teacher model makes predictions on the target domain data to obtain the pseudo-label of the target domain; the student model with the auxiliary prototype classifier uses the pseudo-label of the target domain to perform Training is performed to update parameters; at the same time, during the training process, the feature vectors of different classes of source domain data are saved in the source domain memory bank, and the feature vectors of different classes of target domain data are saved in the target domain memory bank; after the second epoch of training, the teacher model updates its parameters through the exponential moving average of the parameters of the student model; in addition, the source domain memory bank and the target domain memory bank of the same category are spliced, and the splicing results are clustered using the KMeans algorithm to obtain the prototype of each class for auxiliary prototype classifier update; in each subsequent epoch training process, the difference between the training and optimization methods of the student model and the teacher model and the second epoch is that the auxiliary prototype classifier updates the prototype of each class through the exponential moving average of the clustering results of the memory bank until the training is completed.

[0013] To achieve the above-mentioned objectives, the second aspect of the present invention proposes a computer-readable storage medium on which a domain-adaptive remote sensing image semantic segmentation program is stored. When the domain-adaptive remote sensing image semantic segmentation program is executed by a processor, the domain-adaptive remote sensing image semantic segmentation method based on domain adaptation as described above is implemented.

[0014] To achieve the above-mentioned objectives, the third aspect of the present invention proposes a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the domain-adaptive-based remote sensing image semantic segmentation method as described above is implemented.

[0015] To achieve the above-mentioned purpose, the fourth embodiment of the present invention proposes a remote sensing image semantic segmentation device based on domain adaptation, including: an acquisition module, the acquisition module is used to acquire a remote sensing dataset, wherein the remote sensing dataset includes labeled source domain data and unlabeled target domain data; a model construction module, the model construction module is used to construct an average teacher framework with an auxiliary prototype classifier, wherein the average teacher framework with the auxiliary prototype classifier includes a teacher model and a student model; a training module, the training module is used to use the remote sensing dataset to train and optimize the parameters of the student model, wherein the teacher model uses exponential moving average to update its parameters, and the auxiliary prototype classifier uses exponential moving average to update its weights; a semantic segmentation module, the semantic segmentation module is used to input the unlabeled target domain data into the trained student model for point-by-point prediction to obtain the segmentation result corresponding to the unlabeled target domain data.

[0016] According to the domain-adaptive remote sensing image semantic segmentation device of an embodiment of the present invention, by constructing an average teacher framework with an auxiliary prototype classifier, it is possible to solve the domain difference problem in the cross-domain semantic segmentation task of remote sensing images, effectively extract the region of interest from the target domain data, and achieve class hierarchy alignment between the source domain and the target domain, thereby improving the segmentation performance.

[0017] In addition, the remote sensing image semantic segmentation device based on domain adaptation proposed in the above embodiment of the present invention may also have the following additional technical features:

[0018] Optionally, constructing an average teacher framework with an auxiliary prototype classifier includes: the student model includes a feature encoder and a parameterized classifier, and DeepLabV2 is used as the network structure of the student model, and ResNet-101 is used as its skeleton; the network structure and skeleton of the teacher model are consistent with the student model; corresponding categories of memory libraries are constructed for the source domain data and the target domain data in the form of queues, so that after the features output by the corresponding feature encoder are embedded and filtered, they are saved in the memory libraries of the corresponding categories as feature vectors of different categories; the memory libraries of the corresponding categories of the source domain and the target domain are spliced, and the prototype corresponding to the category is clustered using the KMeans clustering algorithm to serve as an auxiliary prototype classifier.

[0019] Optionally, the remote sensing dataset is used to train and optimize the parameters of the student model, including: in the first epoch of training, the student model is trained and parameter optimized using the labeled source domain data, and the feature vectors of different classes of the source domain data are saved in the source domain memory library; after the first epoch of training, the teacher model uses the parameters of the student model to initialize the parameters, and uses the KMeans algorithm to cluster the source domain memory library to obtain the initialized prototype of each class as an auxiliary prototype classifier; in the second epoch of training, the student model with the auxiliary prototype classifier is trained using the labeled source domain data; the teacher model makes predictions on the target domain data to obtain the pseudo-label of the target domain; the student model with the auxiliary prototype classifier is trained using the pseudo-label of the target domain, to update the parameters; at the same time, during the training process, the feature vectors of different classes of source domain data are saved in the source domain memory bank, and the feature vectors of different classes of target domain data are saved in the target domain memory bank; after the second epoch of training, the teacher model updates its parameters by the exponential moving average of the parameters of the student model; in addition, the source domain memory bank and the target domain memory bank of the same category are spliced, and the splicing results are clustered using the KMeans algorithm to obtain the prototype of each class to assist in updating the prototype classifier; in each subsequent epoch training process, the training and optimization methods of the student model and the teacher model are different from those of the second epoch in that the auxiliary prototype classifier updates the prototype of each class by exponentially moving average of the clustering results of the source domain and target domain memory banks until the training is completed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 2 is a flowchart of a remote sensing image semantic segmentation method based on domain adaptation according to an embodiment of the present invention;

[0021] Figure 2 2. A schematic diagram of a model training framework for remote sensing image semantic segmentation based on domain adaptation according to an embodiment of the present invention;

[0022] Figure 3 4 is a block diagram of a remote sensing image semantic segmentation device based on domain adaptation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0024] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0025] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] Figure 1 FIG. 1 is a flow chart of a remote sensing image semantic segmentation method based on domain adaptation according to an embodiment of the present invention. Figure 1 As shown in FIG, the domain adaptation-based remote sensing image semantic segmentation method includes the following steps:

[0027] S101: Acquire a remote sensing dataset, wherein the remote sensing dataset includes labeled source domain data and unlabeled target domain data.

[0028] It should be noted that the source domain data is the red, green, and blue (RGB) band data of Potsdam, and the target domain data is the near-infrared green and blue (IRGB) band data of Vaihingen.

[0029] That is, remote sensing datasets can be downloaded from the Internet.

[0030] S102: Construct an average teacher framework with an auxiliary prototype classifier, wherein the average teacher framework with the auxiliary prototype classifier includes a teacher model and a student model.

[0031] As an embodiment, constructing a Mean Teacher Framework with an Auxiliary prototype classifier (MTA) includes:

[0032] The student model includes a feature encoder and a parameterized classifier, and uses DeepLabV2 as the network structure of the student model and ResNet-101 as its backbone. The network structure and backbone of the teacher model are consistent with those of the student model.

[0033] Build corresponding category memory banks for source domain data and target domain data in the form of queues, so that after the features output by the corresponding feature encoder are embedded and filtered, they are saved in the corresponding category memory banks as feature vectors of different categories;

[0034] It should be noted that the feature vectors extracted by the student model are filtered according to the following embedding filter mechanism, and then the memory bank is updated and deleted:

[0035]

[0036] Among them, <,> represents the inner product, ⊙ represents the Hadamard product; for the source domain samples, For target domain samples,

[0037]

[0038] The memory banks of the corresponding categories in the source domain and the target domain are spliced ​​together, and the prototype corresponding to the category is clustered using the KMeans clustering algorithm to serve as an auxiliary prototype classifier.

[0039] That is to say, if Figure 2 As shown in the figure, the target images are input into the teacher model after weak augmentation, and are input into the student model after strong augmentation. The weak augmentation includes horizontal flipping, vertical flipping, image sharpening, and color jittering, and the strong augmentation includes random rotation, shear projection, and displacement. Both the teacher model and the student model include a feature encoder and a parametric classifier, and use DeepLabV2 as the network structure and ResNet101 as the skeleton. After the weakly enhanced target image is input into the teacher model, the pseudo labels are obtained. After the strongly enhanced target image is input into the student model, the prediction results of the parametric classifier and the auxiliary prototype classifier are respectively compared with the pseudo labels from the teacher model to calculate the cross entropy loss L. t 、 The parameters of the student model are optimized to enhance the classification ability of the target domain data. At the same time, the target features output by the feature encoder are embedded and filtered, and the feature vectors of different categories are saved to the target memory bank of the corresponding category.

[0040] The weakly enhanced source images are input to the student model, and the prediction results of the parameterized classifier and the auxiliary prototype classifier are respectively compared with the labels of the source images to calculate the cross entropy loss L s 、 At the same time, the source features output by the feature encoder are embedded and filtered, and the feature vectors of different categories are saved to the source memory bank (Source MemoryBank) of the corresponding category.

[0041] After the feature vectors of the corresponding categories in the target memory and the source memory are concatenated, they are input into the KMeans clustering algorithm, and the prototype of each class is output. After being updated by the Exponential Moving Average (EMA), it is used as an auxiliary prototype classifier.

[0042] The cross entropy loss is calculated as follows:

[0043]

[0044] Among them, p represents the prediction result of the parameterized classifier or auxiliary prototype classifier of the source domain or target domain, y represents the true label of the source domain or the pseudo label of the target domain, N represents the number of pixels, and K represents the number of categories.

[0045] The optimization objective function of the student model is defined as:

[0046]

[0047] in, In addition, the memory size of each class in the source domain and the target domain is 16384x256; the parameterized classifier consists of a convolutional layer with a convolution kernel size of 1, padding of 0, and a stride of 1.

[0048] The teacher model parameters are updated by the following exponential moving average:

[0049]

[0050] in, represents the parameters of the teacher model after training iteration ι epochs, Represents the parameters of the student model after training iteration ι epochs, ι ≥ 2, smoothing coefficient α = 0.99; when ι = 1, the parameters of the student model are used to initialize the teacher model; auxiliary prototype classifier e K Represents the prototype of the K-th class.

[0051] Category c prototype e c The exponential moving average update method is defined as:

[0052]

[0053] in, Represents the prototype of category c after training iteration I epoch, It represents the class prototype obtained by KMeans clustering of the class memory of the source domain and the target domain after the training iteration epoch I, I≥3; when I=2, e c It is obtained by performing KMeans clustering on the class memory libraries of the source domain and the target domain; when I=1, e c It is obtained by performing KMeans clustering on the class memory library of the source domain.

[0054] S103, using the remote sensing dataset to train and optimize the parameters of the student model, wherein the teacher model uses exponential moving average to update its parameters, and the auxiliary prototype classifier uses exponential moving average to update its weight.

[0055] As an embodiment, a remote sensing dataset is used to train and optimize the parameters of a student model, including: in the first epoch of training, the student model is trained and parameterized using labeled source domain data, and feature vectors of different classes of the source domain data are saved in a source domain memory bank; after the first epoch of training, the teacher model is initialized using the parameters of the student model, and the source domain memory bank is clustered using a KMeans algorithm to obtain an initialized prototype of each class as an auxiliary prototype classifier; in the second epoch of training, the student model with the auxiliary prototype classifier is trained using labeled source domain data; the teacher model makes predictions on the target domain data to obtain pseudo labels of the target domain; the student model with the auxiliary prototype classifier is trained using the pseudo labels of the target domain, to update the parameters; at the same time, during the training process, the feature vectors of different classes of source domain data are saved in the source domain memory bank, and the feature vectors of different classes of target domain data are saved in the target domain memory bank; after the second epoch of training, the teacher model updates its parameters through the exponential moving average of the parameters of the student model; in addition, the source domain memory bank and the target domain memory bank of the same category are spliced, and the splicing results are clustered using the KMeans algorithm to obtain the prototype of each class for auxiliary prototype classifier update; in each subsequent epoch training process, the difference between the training and optimization methods of the student model and the teacher model and the second epoch is that the auxiliary prototype classifier updates the prototype of each class by exponential moving average of the clustering results of the source domain and target domain memory banks until the training is completed.

[0056] It should be noted that stochastic gradient descent (SGD) is used as the optimizer in training, and the weight decay coefficient, impulse value, and initial learning rate are set to 5e-4, 0.9, and 2.5e-4 respectively; the learning rate is gradually reduced using a polynomial decay strategy, and the current learning rate is equal to the initial learning rate multiplied by Among them, power = 0.9.

[0057] S104: input the unlabeled target domain data into the trained student model for point-by-point prediction to obtain a segmentation result corresponding to the unlabeled target domain data.

[0058] In summary, the present invention can solve the domain difference problem in the cross-domain semantic segmentation task of remote sensing images by constructing an average teacher framework with an auxiliary prototype classifier, and can effectively extract regions of interest from target domain data; compared with the multi-stage UDA method, the present invention does not require complex training techniques and multi-stage training strategies; in addition, it can achieve class hierarchy alignment between the source domain and the target domain; compared with other prototype calculation methods that only use the source domain or the target domain, the prototype obtained by clustering in the present invention is more domain-invariant.

[0059] In order to implement the above-mentioned embodiment, an embodiment of the present invention proposes a computer-readable storage medium, on which a domain-adaptive remote sensing image semantic segmentation program is stored. When the domain-adaptive remote sensing image semantic segmentation program is executed by a processor, the domain-adaptive remote sensing image semantic segmentation method based on domain adaptation as described above is implemented.

[0060] According to the computer-readable storage medium of an embodiment of the present invention, by storing a domain-adaptive remote sensing image semantic segmentation program, the processor implements the domain-adaptive remote sensing image semantic segmentation method as described above when executing the domain-adaptive remote sensing image semantic segmentation program. Thus, by constructing an average teacher framework with an auxiliary prototype classifier, the domain difference problem in the cross-domain semantic segmentation task of remote sensing images can be solved, the region of interest can be effectively extracted from the target domain data, and the class hierarchy between the source domain and the target domain can be aligned, thereby effectively improving the segmentation performance on the target domain data.

[0061] In order to implement the above embodiment, an embodiment of the present invention proposes a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements the above-mentioned domain adaptation-based remote sensing image semantic segmentation method.

[0062] According to the computer device of an embodiment of the present invention, the domain-adaptive remote sensing image semantic segmentation program is stored in the memory, so that when the processor executes the domain-adaptive remote sensing image semantic segmentation program, the domain-adaptive remote sensing image semantic segmentation method as described above is implemented. Thus, by constructing an average teacher framework with an auxiliary prototype classifier, the domain difference problem in the cross-domain semantic segmentation task of remote sensing images can be solved, the region of interest can be effectively extracted from the target domain data, and the class hierarchy between the source domain and the target domain can be aligned, thereby effectively improving the segmentation performance on the target domain data.

[0063] In order to implement the above embodiment, the embodiment of the present invention also proposes a remote sensing image semantic segmentation device based on domain adaptation, such as Figure 3 As shown, the domain adaptation-based remote sensing image semantic segmentation device includes: an acquisition module 10, a model construction module 20, a training module 30 and a semantic segmentation module 40.

[0064] Among them, the acquisition module 10 is used to obtain a remote sensing dataset, wherein the remote sensing dataset includes labeled source domain data and unlabeled target domain data; the model construction module 20 is used to construct an average teacher framework with an auxiliary prototype classifier, wherein the average teacher framework with the auxiliary prototype classifier includes a teacher model and a student model; the training module 30 is used to train and optimize the parameters of the student model using the remote sensing dataset, wherein the teacher model uses an exponential moving average to update its parameters, and the auxiliary prototype classifier uses an exponential moving average to update its weights; the semantic segmentation module 40 is used to input the unlabeled target domain data into the trained student model for point-by-point prediction to obtain the segmentation result corresponding to the unlabeled target domain data.

[0065] It should be noted that the above description and examples of the remote sensing image semantic segmentation method based on domain adaptation are also applicable to the remote sensing image semantic segmentation device based on domain adaptation in this embodiment, and will not be repeated here.

[0066] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0070] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third etc. does not indicate any order. These words may be interpreted as names.

[0071] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0072] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

[0073] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0074] In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connect," "fixed," etc. should be understood broadly. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0075] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0076] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0077] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A remote sensing image semantic segmentation method based on domain adaptation, characterized in that: The following steps are involved: Acquiring a remote sensing dataset, wherein the remote sensing dataset includes labeled source domain data and unlabeled target domain data; Constructing an average teacher framework with an auxiliary prototype classifier, wherein the average teacher framework with the auxiliary prototype classifier includes a teacher model and a student model; Using the remote sensing dataset to train and optimize the parameters of the student model, wherein the teacher model updates its parameters using exponential moving average and the auxiliary prototype classifier updates its weights using exponential moving average; Inputting the unlabeled target domain data into the trained student model for point-by-point prediction to obtain a segmentation result corresponding to the unlabeled target domain data; Among them, building an average teacher framework with auxiliary prototype classifiers includes: The student model includes a feature encoder and a parameterized classifier, and uses DeepLabV2 as the network structure of the student model and ResNet-101 as its skeleton; the network structure and skeleton of the teacher model are consistent with those of the student model; Constructing corresponding category memory banks for the source domain data and the target domain data in the form of queues, respectively, so that features output by the corresponding feature encoders are stored in the corresponding category memory banks as feature vectors of different categories after being embedded and filtered; The memory banks of the corresponding categories of the source domain and the target domain are spliced ​​together, and the prototype corresponding to the category is clustered using the KMeans clustering algorithm to serve as an auxiliary prototype classifier; The remote sensing data set is used to train and optimize the parameters of the student model, including: In the first epoch of training, the student model is trained and parameter optimized using the labeled source domain data, and the feature vectors of different classes of the source domain data are saved in the source domain memory; After the first epoch of training, the teacher model uses the parameters of the student model to initialize parameters, and uses the KMeans algorithm to cluster the source domain memory library to obtain the initialized prototype of each class as an auxiliary prototype classifier; In the second epoch of training, the student model with the auxiliary prototype classifier is trained using the labeled source domain data; the teacher model makes predictions on the target domain data to obtain pseudo-labels of the target domain; the student model with the auxiliary prototype classifier is trained using the pseudo-labels of the target domain to update its parameters; and during the training process, the feature vectors of different classes of the source domain data are saved to the source domain memory bank, and the feature vectors of different classes of the target domain data are saved to the target domain memory bank; After the second epoch of training, the teacher model updates its parameters by taking the exponential moving average of the student model’s parameters. In addition, the source domain memory and target domain memory of the same category are concatenated, and the concatenation results are clustered using the KMeans algorithm to obtain the prototype of each class for auxiliary prototype classifier update. In each subsequent epoch training process, the training and optimization methods of the student model and the teacher model are different from those of the second epoch in that the auxiliary prototype classifier updates the prototype of each class by clustering the memory banks of the source domain and the target domain in an exponential moving average manner until the training is completed.

2. A computer-readable storage medium, characterized in that A remote sensing image semantic segmentation program based on domain adaptation is stored thereon, and when the remote sensing image semantic segmentation program based on domain adaptation is executed by a processor, the remote sensing image semantic segmentation method based on domain adaptation as claimed in claim 1 is implemented.

3. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the remote sensing image semantic segmentation method based on domain adaptation as claimed in claim 1 is implemented.

4. A remote sensing image semantic segmentation device based on domain adaptation, characterized in that: include: An acquisition module, configured to acquire a remote sensing data set, wherein the remote sensing data set includes labeled source domain data and unlabeled target domain data; A model construction module, wherein the model construction module is used to construct an average teacher framework with an auxiliary prototype classifier, wherein the average teacher framework with the auxiliary prototype classifier includes a teacher model and a student model; a training module, the training module being configured to train and optimize parameters of the student model using the remote sensing dataset, wherein the teacher model updates its parameters using an exponential moving average, and the auxiliary prototype classifier updates its weights using an exponential moving average; A semantic segmentation module, configured to input the unlabeled target domain data into a trained student model for point-by-point prediction to obtain a segmentation result corresponding to the unlabeled target domain data; Among them, building an average teacher framework with auxiliary prototype classifiers includes: The student model includes a feature encoder and a parameterized classifier, and uses DeepLabV2 as the network structure of the student model and ResNet-101 as its skeleton; the network structure and skeleton of the teacher model are consistent with those of the student model; Constructing corresponding category memory banks for the source domain data and the target domain data in the form of queues, respectively, so that features output by the corresponding feature encoders are stored in the corresponding category memory banks as feature vectors of different categories after being embedded and filtered; The memory banks of the corresponding categories of the source domain and the target domain are spliced ​​together, and the prototype corresponding to the category is clustered using the KMeans clustering algorithm to serve as an auxiliary prototype classifier; The remote sensing data set is used to train and optimize the parameters of the student model, including: In the first epoch of training, the student model is trained and parameter optimized using the labeled source domain data, and the feature vectors of different classes of the source domain data are saved in the source domain memory; After the first epoch of training, the teacher model uses the parameters of the student model to initialize parameters, and uses the KMeans algorithm to cluster the source domain memory library to obtain the initialized prototype of each class as an auxiliary prototype classifier; In the second epoch of training, the student model with the auxiliary prototype classifier is trained using the labeled source domain data; the teacher model makes predictions on the target domain data to obtain pseudo-labels of the target domain; the student model with the auxiliary prototype classifier is trained using the pseudo-labels of the target domain to update its parameters; and during the training process, the feature vectors of different classes of the source domain data are saved to the source domain memory bank, and the feature vectors of different classes of the target domain data are saved to the target domain memory bank; After the second epoch of training, the teacher model updates its parameters by taking the exponential moving average of the student model’s parameters. In addition, the source domain memory and target domain memory of the same category are concatenated, and the concatenation results are clustered using the KMeans algorithm to obtain the prototype of each class for auxiliary prototype classifier update. In each subsequent epoch training process, the training and optimization methods of the student model and the teacher model are different from those of the second epoch in that the auxiliary prototype classifier updates the prototype of each class in an exponential moving average manner by clustering the results of the memory banks of the source domain and the target domain until the training is completed.

Images