A remote sensing semantic segmentation method and system based on domain persistence learning

Abstract

The application discloses a remote sensing semantic segmentation method and system based on domain persistence learning. The steps are as follows: data preprocessing is performed on a remote sensing domain sequence composed of multiple remote sensing domains to form a remote sensing domain sequence dataset; a cross-domain persistence learning model is constructed as a semantic segmentation network; the cross-domain persistence learning model is trained in a remote sensing domain in sequence, a loss function based on a combination of a class filtering and a feature space knowledge distillation mode is adopted to relieve catastrophic forgetting in the training process of the cross-domain persistence learning model, and a trained cross-domain persistence learning model is obtained; the cross-domain persistence learning model is trained in multiple remote sensing domains in sequence in an incremental training test mode, training data only contains current remote sensing domain data, and testing is performed on all past accumulated remote sensing domains; after training of all remote sensing domains is completed, the domain persistence learning model is embedded into a remote sensing intelligent interpretation device, and finally, segmentation results of a multi-source remote sensing domain sequence are obtained.

Description

Technical Field

[0001] This invention relates to a remote sensing semantic segmentation method and system based on domain continuous learning, belonging to the fields of remote sensing intelligent interpretation, remote sensing natural resource monitoring, and forest fire detection. It can be used as a dynamic and general semantic segmentation technology and extended to downstream tasks such as remote sensing natural resource monitoring and fire smoke monitoring in multi-source remote sensing domain continuous learning scenarios. Background Technology

[0002] The rapid development of deep learning technology has provided new support for intelligent remote sensing interpretation. Among these advancements, deep learning-based remote sensing semantic segmentation methods have been proposed and applied in fields such as environmental monitoring, vegetation classification, and urban planning. However, existing deep learning-based remote sensing semantic segmentation methods primarily rely on the in-distribution assumption and do not consider updates to the model based on out-of-domain data. In reality, the characteristics of satellite revisits, multi-class sensors, and full land cover all pose new requirements for extending remote sensing semantic segmentation models to multi-source domain remote sensing scenarios. Developing a general semantic segmentation method applicable to multi-source domain remote sensing scenarios is more likely to meet the needs of practical applications.

[0003] However, directly extending deep learning-based semantic segmentation methods to different remote sensing domains leads to a sharp decline in performance, a phenomenon known as the catastrophic forgetting problem. To address this, persistent learning (also known as incremental learning) methods have emerged, encompassing three scenarios: task-oriented incremental learning, category-oriented incremental learning, and domain-oriented incremental learning. In natural images, persistent learning methods for semantic segmentation tasks have been proposed. For example, Cermelli et al. proposed a category-oriented incremental method based on knowledge distillation. Compared to natural images, persistent learning methods for remote sensing semantic segmentation tasks are a relatively new field with limited research. Specifically, Tasar et al. first investigated incremental learning scenarios for remote sensing segmentation, and Shan et al. designed two effective modules embedded in their proposed category-oriented incremental segmentation framework, eliminating the need to access previous data. However, these experimental settings invariably only considered category-oriented incremental research within the same remote sensing domain, neglecting the need for domain-oriented incremental learning scenarios. This has spurred research into semantic segmentation tasks within domain-oriented incremental learning scenarios. Summary of the Invention

[0004] This invention addresses the problem of catastrophic forgetting in intelligent remote sensing interpretation applications by providing a remote sensing semantic segmentation method and system based on domain-based continuous learning. This enables the remote sensing semantic segmentation model to continuously learn from multiple remote sensing domains. It can be used as a dynamic semantic segmentation model that can be scaled over time for routine monitoring of remote sensing natural resources, or as a general semantic segmentation model extended to multi-source remote sensing data.

[0005] To solve the above technical problems, the present invention adopts the following solution:

[0006] A remote sensing semantic segmentation method based on domain-persistent learning includes the following steps:

[0007] Step 1: Perform data preprocessing on the remote sensing domain sequence composed of multiple remote sensing domains to form a remote sensing domain sequence dataset; the preprocessing includes unified operations of data augmentation, image cropping, and label conversion; the dataset includes a training set and a test set;

[0008] Step 2: Construct a cross-domain continuous learning model, with semantic segmentation as the downstream task;

[0009] Step 3: Based on the training set from Step 1, the cross-domain persistent learning model from Step 2 is trained sequentially in a certain remote sensing domain. A combination of category-based selection and feature space-based knowledge distillation is used as the loss function to mitigate catastrophic forgetting during the training process of the cross-domain persistent learning model, resulting in the trained cross-domain persistent learning model.

[0010] Step 4: Following the incremental training and testing approach, the cross-domain continuous learning model is trained sequentially in multiple remote sensing domains. The training data only includes data from the current remote sensing domain. The model is then tested on all previously accumulated remote sensing domains. After training all remote sensing domain sequences, the cross-domain continuous learning model is embedded into the remote sensing intelligent interpretation device to obtain the segmentation results of the multi-source remote sensing domain sequences.

[0011] In step 2, the structure of the cross-domain persistent learning model consists of a shared encoder. and K domain-specific decoders The system comprises a lightweight semantic segmentation ERFNet model with a shared encoder. Its core structure seamlessly integrates multiple Domain Residual Adaptive (DRA) modules into the ERFNet model. The ERFNet model's structure is based on an improved ResNet, replacing the standard 3×3 convolutional layers with a stack of 3×1 and 1×3 convolutional layers. The decoder structure consists of K parallel decoders, identical to those in the ERFNet model. The number of decoders is the same as the number of remote sensing domains. Decoders corresponding to different remote sensing domains are selected during training and testing.

[0012] For each remote sensing domain D k =(x k ,y k x in the data k ,y k These are remote sensing images and their corresponding semantic labels, used in the training of the cross-domain continuous learning model. k ,y k These are the truth values ​​of the input and the corresponding output, respectively. Represented as predicted values; Cross-domain continuous learning model M k The learning process in the k-th domain is represented as:

[0013]

[0014] For cross-domain continuous learning model M k Perform parameter decomposition, using This refers to domain-specific parameters, which are parameters corresponding to the unique structures of different remote sensing domains. This is represented as a domain-shared parameter, which corresponds to the parameters shared by different remote sensing domains in the model structure. The model structure is an encoder-decoder structure, which can be further expressed as:

[0015]

[0016] in, Decompose into α k c is a parameter specific to the encoder ε domain. k The corresponding decoder has domain-specific parameters;

[0017] The structure of the Domain Residual Adaptive Module (DRA):

[0018] The Domain Residual Adaptive (DRA) module reparameterizes domain-specific and domain-shared parameters through model reparameterization, extracting and training domain-shared and domain-specific features respectively. Specifically, the features extracted in the j-th module are: Features introduced by the domain residual adaptive module Connect the features passed in from the previous module j-1 in a hierarchical manner. composition,

[0019]

[0020] Among them, features Including the domain shared portion g j (·) and domain-specific parts The two features are combined, and the structure is as follows:

[0021]

[0022] Among them, g j It is a domain-shared structure, consisting of 3×1 and 1×3 convolutional layers, followed by the ReLU activation function; and It is a residual adaptive structure unique to parallel domains. It is a 1×1 convolutional layer. This represents a batch normalization layer, where each domain has a unique domain-specific structure. and The domains are combined in parallel, that is, the features output by the previous module are combined. Input to the domain shared structure g respectively j and the domain-specific structure corresponding to the current domain The two features are then input again into the domain-specific structure corresponding to the current domain. Obtain features

[0023] In step 3, the loss function that combines category filtering and feature space knowledge distillation consists of a category filtering distillation function, a feature space distillation function, and a cross-entropy function.

[0024] (1) Distillation function L based on category selection D

[0025] The cross-domain persistent learning model is trained sequentially on multiple domains. The model is then trained on the k-th domain. During training, it is represented as M. k Current model M k Domain shared parameter W s Initialization based on past cross-domain continuous learning model M k-1 Domain-shared parameters; L D The distillation process will convert the current domain data x into... k Input into the cross-domain continuous learning model M corresponding to the current domain k The cross-domain continuous learning model M corresponding to the past domain k-1 In this process, the predicted outputs of the cross-domain continuous learning model are distilled sequentially across multiple domains, and the knowledge distillation loss function L is based on category selection. D for:

[0026]

[0027]

[0028]

[0029] in, and M represents the cross-domain continuous learning model corresponding to the current domain. k The cross-domain continuous learning model M corresponding to the past domain k-1 For input x respectively k The predicted output, This is expressed as the distillation loss function over the past domain, and sequentially over all past domains. Distillation is performed on the domain; the knowledge distillation function based on category selection only performs distillation in the current domain. and past domain Knowledge distillation is performed in the common classes, and calculations are performed on the spatial and channel dimensions respectively;

[0030]

[0031] Among them, μ k ∈{0,1} C×H×W represents the pixel-level binary label, obtained through one-hot encoding, C μ represents the current domain and the common classes of the past domain D i , 0 < i < k, N k represents all pixels in the current domain;

[0032] (2) Feature space distillation function L F

[0033] The distillation function L of the feature space F is:

[0034]

[0035]

[0036]

[0037] Among them, and represent the outputs of the cross-domain continuous learning model M k corresponding to the current domain and the cross-domain continuous learning model M k-1 corresponding to the past domain in the feature space, where represents the output of the encoder in the feature space, where represents the encoder k corresponding to the input x in the output features of domain k - 1. ‖·‖ represents the Euclidean norm function, similar to the knowledge distillation loss function based on class filtering, and distillation is performed sequentially on all past domains

[0038] (3) Cross-entropy loss function L CE

[0039] The cross-entropy loss function is:

[0040]

[0041] Among them, represents the output of the current model, ψ k represents the cross-entropy function;

[0042] (4) The overall loss is the sum of various loss functions:

[0043] L total =λ1·L CE +λ2·L D +λ3·L F

[0044] Where λ1, λ2, and λ3 represent the cross-entropy function L, respectively. CE Special category distillation function L D and the characteristic space distillation function L F The weights;

[0045] (5) Optimization Strategy

[0046] Cross-domain continuous learning model in the k-th domain During training, the optimization strategy for the cross-domain continuous learning model is as follows: for domain-specific parameters... Domain shared parameters Different optimization strategies are used in some areas. Parameter initialization is based on The decoder's classification layer uses random initialization; furthermore, it freezes all fields except the current one. Domain-specific parameters corresponding to other domains Domain shared parameters Initialization based on M k-1 Corresponding shared parameters

[0047] In step 4, the cross-domain continuous learning model is trained sequentially in multiple remote sensing domains using an incremental training and testing approach, as follows:

[0048] Step 1: Based on the current domain k, D k The new data and the cross-domain continuous learning model M corresponding to the previous domain k-1 k-1 ;

[0049] Step 2: Initialization, by adding a new domain-specific layer w k To model M k-1 Get M k Based on M k-1 w in k-1 Parameter initialization M k w in k Simultaneously, freeze the domain-specific parameters corresponding to all domains except the current domain.

[0050] Step 3: Iterative training, for each batch of data in each iteration:

[0051] The cross-domain continuous learning model M corresponding to the current domain k k (x k ,k) Forward propagation through parameter w k Calculate L CE ;

[0052] Backpropagation model M k (x k (k-1) via parameter w k-1 Backpropagation model M k-1 (x k (k-1) via parameter w k-1 Calculate the distillation loss L for both models. D and L F ;

[0053] Comprehensive calculation of loss L total =λ1·L CE +λ2·L D +λ3·L F ;

[0054] Update the learning rate, where the domain-shared portion w s and domain-specific parts w k Update using different learning rates;

[0055] Cross-domain continuous learning model M k After training is completed in the current domain k, testing is performed on all past domains.

[0056] This invention presents a remote sensing semantic segmentation system based on domain-based continuous learning. Aiming at applications such as remote sensing natural element segmentation and forest fire smoke segmentation, it proposes a remote sensing semantic segmentation model system with embedded domain-based continuous learning capabilities, characterized by its dynamic, universal, and sustainable nature. This system can continuously learn from data from multiple remote sensing domains, enabling it to achieve high performance on the current remote sensing domain task while maintaining high performance across all past remote sensing domain tasks.

[0057] This invention includes: a data acquisition module, a data preprocessing module, a cross-domain continuous learning model module, and a training module;

[0058] The data acquisition module acquires multi-source remote sensing data consisting of multiple remote sensing domains from different remote sensing sensor platforms, forming a remote sensing domain sequence;

[0059] The data preprocessing module, based on the remote sensing domain sequence obtained from the data acquisition module, performs further data preprocessing operations. Through data augmentation, image cropping, and label conversion, remote sensing data with the same format and remote sensing category labels are obtained. The remote sensing data is then divided into training and test sets according to a certain ratio.

[0060] The cross-domain persistent learning model module constructs a cross-domain persistent learning model for semantic segmentation tasks in the remote sensing domain. This model consists of a shared encoder and parallel decoders (corresponding to the number of domain sequences). The shared encoder is based on the lightweight semantic segmentation ERFNet model, with a core structure of multiple Domain Residual Adaptive Modules (DRAs) seamlessly integrated into the ERFNet model. The DRAs extract and train domain-shared and domain-specific features by reparameterizing domain-specific and domain-shared parameters. The cross-domain persistent learning model reduces the number of parameters required for model expansion by sharing parameters, while ensuring that the model maintains its performance in past remote sensing domains when learning new ones. Furthermore, the cross-domain persistent learning model embeds a loss function combining category selection and feature space knowledge distillation, along with a cross-entropy loss function, to ensure the training of the cross-domain persistent learning model in the persistent training module.

[0061] The continuous training module trains the cross-domain continuous learning model sequentially across multiple remote sensing domains processed by the data preprocessing module, using an incremental training and testing approach. The training data contains only data from the current remote sensing domain, and the model is tested on all previously accumulated remote sensing domains. After training all remote sensing domain sequences, the cross-domain continuous learning model is embedded into the remote sensing intelligent interpretation device, ultimately obtaining the segmentation results of the multi-source remote sensing domain sequences. Specifically, when training the model in the current domain, it first uses new data from the current domain and the cross-domain continuous learning model corresponding to the previous domain (i.e., past domains). The next initialization process... The cross-domain persistent learning model for the current domain is obtained by adding new domain-specific parameters to the cross-domain persistent learning model corresponding to the previous domain (i.e., past domains). Domain-shared and domain-specific parameters for the current domain are initialized, while domain-specific parameters for all domains except the current domain are frozen. Iterative training is then performed. For each batch of data in each iteration of the current domain, the cross-domain persistent learning model for the current domain is forward-propagated and backward-propagated. The cross-entropy function and distillation function are calculated using the domain-specific and domain-shared parameters, respectively, and the learning rate is updated. Finally, after the cross-domain persistent learning model is trained in the current domain, it is tested on all past domains.

[0062] The advantages of this invention compared to the prior art are:

[0063] Existing deep learning-based remote sensing semantic segmentation methods are mostly based on static settings. This means that when extended to multi-source data from different times, regions, and sensors, catastrophic forgetting occurs, leading to a significant drop in model monitoring performance. This invention aims to provide reasonable support for tasks such as remote sensing natural resource monitoring. The invention's advantages are reflected in:

[0064] First, this invention explores a remote sensing continuous learning method from the perspective of domain increment for the first time, setting a more suitable scenario for real-world remote sensing applications compared to existing remote sensing continuous learning techniques. This invention's remote sensing semantic segmentation method based on domain continuous learning is designed for multi-source remote sensing data, enabling continuous learning of semantic segmentation methods across multiple remote sensing domains. Existing remote sensing continuous learning techniques are not suitable for domain increment tasks and still suffer from catastrophic forgetting.

[0065] Second, the downstream task of the method in this invention is pixel-level semantic segmentation, which provides more accurate monitoring results compared to image-level land cover classification. While most existing remote sensing continuous learning techniques achieve continuous learning in image classification tasks, these techniques can only output image-level land cover category labels. Considering the complex and diverse nature of remote sensing land cover types, assigning only one category to an entire image often results in significant errors. This invention constructs a cross-domain continuous learning model with semantic segmentation as the downstream task, enabling continuous learning across multiple remote sensing domains and outputting pixel-level land cover category labels across these domains.

[0066] Third, the method of this invention, based on parameter isolation, is more suitable for remote sensing applications and can be easily and robustly extended to remote sensing applications. Specifically, compared with existing sustainable learning techniques based on data playback, it overcomes the limitations of data storage and data privacy preservation; compared with regularization-based sustainable learning techniques, this invention achieves globally optimal performance and outperforms regularization-based sustainable learning techniques; this invention uses a domain residual adaptive module to embed the sustainable learning model, and employs a distillation function that combines category selection and feature space knowledge distillation, which greatly reduces the increase in parameters of the model in cross-domain sustainable learning, while also maximizing the retention of knowledge learned by the model in past domains.

[0067] Fourth, the remote sensing semantic segmentation system based on domain-based continuous learning developed in this invention can be embedded into remote sensing intelligent interpretation equipment, which can be used to improve the generalization performance of remote sensing intelligent interpretation models extended to multi-source remote sensing data. The downstream task in this embodiment is a semantic segmentation task based on natural resource monitoring, which better extracts eight land cover categories—water bodies, buildings, cultivated land, forest land, grassland, transportation, wasteland, background, and others—from the multi-source remote sensing domain. This invention can also be extended to downstream tasks such as smoke segmentation in forest fire monitoring, serving as a means of routine forest fire risk monitoring. Attached Figure Description

[0068] Figure 1 The dataset is categorized by class.

[0069] Figure 2 , 3 Example dataset;

[0070] Figure 4 For experimental setup;

[0071] Figure 5 This is a structural diagram of the model of the present invention;

[0072] Figure 6 This is a structural diagram of the residual adaptive module;

[0073] Figure 7 Distillation loss function for a specific category;

[0074] Figure 8 This is a flowchart illustrating the implementation of the method of the present invention. Detailed Implementation

[0075] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The examples described below are only some embodiments of the present invention, and not all embodiments.

[0076] like Figure 8 As shown, the embodiments of the present invention mainly include the following steps:

[0077] 1) Multi-source remote sensing data preprocessing

[0078] In this invention, the dataset selection is based on the idea of ​​mining existing data. Representative remote sensing data from various platforms, payload imaging mechanisms, and spatiotemporal spectral resolutions are chosen from a large pool of existing open-source remote sensing datasets. Compared to current remote sensing application models based on deep learning methods, which are mostly tested only on their own self-constructed remote sensing datasets, the data selection in this invention facilitates later evaluation of the generalization performance of deep learning models on existing open-source remote sensing data.

[0079] The dataset for this invention is derived from multi-source remote sensing data, primarily including data from multiple satellites (Gaofen-2, Gaofen-1, Gaofen-6, and WorldView-2 satellites) and aerial imagery from unmanned aerial vehicles (UAVs). The experimental data covers typical rural and urban landscapes across multiple countries and regions. The dataset includes eight land cover categories: water bodies, buildings, cultivated land, forest land, grassland, transportation, wasteland, background, and others. Key information about the dataset is shown in Table 1. The proportion of each category in each dataset is also shown below. Figure 1 As shown, GID, BDCI (2020), deepglobe, LoveDA-urban, and LoveDA-rural are remote sensing datasets of the present invention, representing multiple remote sensing domains. The figure shows the category proportion of each dataset in eight land cover categories: water, buildings, cultivated land, forest land, grassland, transportation, wasteland, background, and others. It can be seen from the figure that the category distribution of each remote sensing domain has obvious differences.

[0080] Table 1. Main information about the experimental dataset

[0081]

[0082] Because the image sizes and label formats vary across datasets, uniform preprocessing, including cropping and label conversion, is performed on all datasets. Each image set includes pixel labels, color labels, and image data, as shown in Table 2. Example datasets are provided below. Figure 2 , 3 As shown. Figure 2 Each column of images represents a different remote sensing dataset, and each row represents a different remote sensing land cover category. Figure 3 Each row in the table represents a different remote sensing dataset. Figure 3 The first and third columns represent different remote sensing image examples, while the second and fourth columns represent the labels corresponding to the remote sensing images. Figure 2 , 3 It can demonstrate the differences in characteristics of different land cover categories in different remote sensing domains.

[0083] Table 2 Dataset Category and Label Information

[0084]

[0085] 2) Technical Solution

[0086] Multi-source remote sensing data collected by different sensors at different times and in different regions can all be considered as multi-source domains of remote sensing. This invention selects the representative dataset GID-BDCI-deepglobe-loveDA_urban-loveDA_rural as the experimental domain sequence, which involves multiple types of sensors and various regions. In the domain-based persistent learning setting, it is assumed that there are N tasks, corresponding to N training domains. Where x k ,y k These represent the domain samples and their corresponding labels. Considering the three mainstream experimental protocol settings in incremental semantic segmentation: sequential, disjoint, and overlapping, the sequential protocol was chosen as the experimental protocol, where the label for each domain is independent. In remote sensing, the definition of the background class is well-defined, so there is no need to consider the changes in the background class in the domain sequence; instead, the labels of the original dataset are used as the labels for the redefined remote sensing domain sequence. Furthermore, and D k-1 There are domain shifts and label space shifts, y k Compared to y k-1 There are common classes and additional new classes, and vice versa.

[0087] like Figure 4 The diagram illustrates the overall training process of the domain-incremental semantic segmentation model. The semantic segmentation model is continuously trained on remote sensing domain sequences. In the k-th step of training, in the domain... Training the model At this point, all past domain data Not available. The goal is for the model to progressively learn each new domain without forgetting knowledge from past domains. After training in the current domain, evaluate the model's performance across all current and past domains. The performance of the k-th step. For simplicity, the domain, dataset, and task are all represented as...

[0088] 3) Model Structure

[0089] like Figure 5 As shown, the remote sensing domain continuous learning architecture consists of two components: a shared encoder. and K domain-specific decoders This shared encoder is based on the lightweight semantic segmentation model ERFNet and consists of multiple Domain Residual Adaptive Modules (DRAs). Each DRA learns both domain-shared and domain-specific features. For each remote sensing domain D... k =(x k ,y k x in the data k ,y k These are remote sensing images and their corresponding semantic labels, used in the training of the cross-domain continuous learning model. k ,y k These are the truth values ​​of the input and the corresponding output, respectively. Represented as predicted values; Cross-domain continuous learning model M k The learning process can be represented as:

[0090]

[0091] For cross-domain continuous learning model M k Perform parameter decomposition, using This refers to domain-specific parameters, which are parameters corresponding to the unique structures of different remote sensing domains. This is represented as a domain-shared parameter, which corresponds to the parameters shared by different remote sensing domains in the model structure. The model structure is an encoder-decoder structure, which can be further expressed as:

[0092]

[0093] in, Decompose into α k c is a domain-specific parameter of the encoder. k The corresponding decoder has domain-specific parameters;

[0094] like Figure 6 As shown, the domain residual adaptive module is a key structure of the model, used to reparameterize the network with both domain-specific and domain-shared parameters. The features in the j-th module are defined as... Features introduced by the domain residual adaptive module The hierarchical connection connects to the features passed in from the previous module. composition,

[0095]

[0096] Specifically, It is a domain shared part g j (·) and domain-specific parts It is connected in series.

[0097]

[0098] Among them, g j It is a domain-shared structure that spans all domains. and It is a residual adaptive structure unique to parallel domains. Where, g j It consists of [3×1] and [1×3] convolutional layers, followed by a ReLU activation function. For domain-specific structures, A [1×1] convolutional layer is represented in the diagram as k parallel structures. Furthermore, This represents a batch regularization layer, which is also a domain-specific structure for parallel processing.

[0099] 4) Loss function and optimization strategy

[0100] The loss function of this model mainly consists of three types of loss functions: a category-specific distillation function, a feature space distillation function, and a cross-entropy function.

[0101] Domain shift is a major challenge when models learn a domain incrementally, while label space shifts also exist between regions that do not overlap in the label space. In recent continuous learning semantic segmentation tasks, researchers have focused on the semantic shift of the background class, i.e., classifying the old class from the previous step as the background class in each step, with all classes belonging to the same domain in their settings. Although these studies do not address the domain shift problem, the approach of using knowledge distillation strategies to solve semantic shifts is worth learning from. However, simply applying the classic knowledge distillation loss function is insufficient to meet the problem setting addressed in this invention. Considering the simultaneous existence of domain shift and label space shift, the model in this invention addresses this problem by introducing knowledge distillation functions at multiple scales and different optimization strategies for domain-specific and domain-shared parameters.

[0102] 1. Distillation function L based on category selection D

[0103] The cross-domain persistent learning model is trained sequentially on multiple domains. The model is then trained on the k-th domain. During training, it is represented as M. k Current model Mk Domain sharing parameter W s Initialize the cross-domain persistent learning model M based on the past k-1 Domain sharing parameter; L D In the distillation process, the current domain data x k Is input into the cross-domain persistent learning model M corresponding to the current domain k And the cross-domain persistent learning model M corresponding to the past domain k-1 Among them, the prediction outputs of the cross-domain persistent learning model in multiple domains are distilled in sequence, and the distillation function L based on class screening D Is:

[0104]

[0105]

[0106]

[0107] Where And Represent the cross-domain persistent learning model M corresponding to the current domain k And the cross-domain persistent learning model M corresponding to the past domain k-1 The prediction outputs for the input x k Respectively, Represents the distillation loss function for the past domain, and distillation is performed sequentially on all past domains The distillation function based on class screening only performs knowledge distillation in the common classes of the current domain And the past domain And calculates separately in terms of spatial and channel dimensions;

[0108]

[0109] Where, μ k ∈{0,1} C×H×W Represents the pixel-level binary label, obtained by one-hot encoding Of, C μ Represents the current domain And the common classes of the past domain D i , 0 < i < k, N k Represents all pixels in the current domain;

[0110] Figure 7 Is the knowledge distillation loss function L for the above class screening D Schematic diagram, as shown in the figure, after calculating And According to the above introduction, class screening and calculation are performed separately from the pixel level and the channel level.

[0111] 2. Characteristic space distillation function L F

[0112] Distillation function L in the characteristic space F for:

[0113]

[0114]

[0115]

[0116] in, and M represents the cross-domain continuous learning model corresponding to the current domain. k The cross-domain continuous learning model M corresponding to the past domain k-1 encoder The output in the feature space, where Represented as encoder Corresponding input x k The output features in domain k-1. ‖·‖ represents the L2 norm function, similar to the knowledge distillation loss function based on category selection, successively applied across all past domains. Distillation is carried out on top;

[0117] 3. Cross-entropy loss function L CE

[0118] The cross-entropy loss function is:

[0119]

[0120] in, Represented as the output of the current model, ψ k Represented as the cross-entropy function;

[0121] 4. The overall loss is a summary of all loss functions:

[0122] L total =λ1·L CE +λ2·L D +λ3·L F

[0123] Where λ1, λ2, and λ3 represent the cross-entropy function L, respectively. CE Special category distillation function L D and the characteristic space distillation function L F The weights;

[0124] 5. Optimization Strategy

[0125] Cross-domain continuous learning model in the k-th domain During training, the optimization strategy for the cross-domain continuous learning model is as follows: for domain-specific parameters... Domain shared parameters Different optimization strategies are used in some areas. Parameter initialization is based on The decoder's classification layer uses random initialization; furthermore, it freezes all fields except the current one. Domain-specific parameters corresponding to other domains Domain shared parameters Initialization based on M k-1 Corresponding shared parameters

[0126] In step 4, the cross-domain continuous learning model is trained sequentially in multiple remote sensing domains using an incremental training and testing approach, as follows:

[0127] Step 1: Based on the current domain k, D k The new data and the cross-domain continuous learning model M corresponding to the previous domain k-1 k-1 ;

[0128] Step 2: Initialization, by adding a new domain-specific layer w k To model M k-1 Get M k Based on M k-1 w in k-1 Parameter initialization M k w in k Simultaneously, freeze the domain-specific parameters corresponding to all domains except the current domain.

[0129] Step 3: Iterative training, for each batch of data in each iteration:

[0130] The cross-domain continuous learning model M corresponding to the current domain k k (x k ,k) Forward propagation through parameter w k Calculate L CE ;

[0131] Backpropagation model M k (x k (k-1) via parameter w k-1 Backpropagation model M k-1 (x k (k-1) via parameter w k-1 Calculate the distillation loss L for both models. D and L F ;

[0132] Comprehensive calculation of loss L total =λ1·L CE +λ2·L D +λ3·LF ;

[0133] Update the learning rate, where the domain-shared portion w s and domain-specific parts w k Update using different learning rates;

[0134] Cross-domain continuous learning model M k After training is completed in the current domain k, testing is performed on all past domains.

[0135] While specific implementation methods of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples. Various changes or modifications can be made to these implementation methods without departing from the principles and implementation of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims.

Claims

1. A remote sensing semantic segmentation method based on domain-based persistent learning, comprising the following steps: Step 1: Perform data preprocessing on the remote sensing domain sequence composed of multiple remote sensing domains to form a remote sensing domain sequence dataset; The preprocessing includes unified operations of data augmentation, image cropping, and label conversion; the dataset includes a training set and a test set. Step 2: Construct a cross-domain continuous learning model as a semantic segmentation network; Step 3: Based on the training set from Step 1, the cross-domain persistent learning model from Step 2 is trained sequentially in a certain remote sensing domain. A combination of category-based selection and feature space-based knowledge distillation is used as the loss function to mitigate catastrophic forgetting during the training process of the cross-domain persistent learning model, resulting in the trained cross-domain persistent learning model. Step 4: Following the incremental training and testing approach, the cross-domain continuous learning model is trained sequentially in multiple remote sensing domains. The training data only contains data from the current remote sensing domain. The model is tested on all previously accumulated remote sensing domains. After training all remote sensing domain sequences, the cross-domain continuous learning model is embedded into the remote sensing intelligent interpretation device to finally obtain the segmentation results of the multi-source remote sensing domain sequences. In step 2, the structure of the cross-domain persistent learning model consists of a shared encoder. and K domain-specific decoders Composition; The shared encoder core structure is to seamlessly add multiple domain residual adaptive modules (DRA) to the ERFNet model. The structure of the ERFNet model is based on the improved ResNet, which replaces the standard 3×3 convolutional layer with a stack of 3×1 and 1×3 convolutional layers. The decoder structure consists of K parallel decoders, which is the same as the decoder structure in the ERFNet model. The number of decoders is the same as the number of remote sensing domains. When training and testing different remote sensing domains, the decoder corresponding to the domain is selected. For each remote sensing domain In the data These are remote sensing images and their corresponding semantic labels, used in the training of a cross-domain continuous learning model. These are the truth values ​​of the input and the corresponding output, respectively. Represented as predicted values; cross-domain continuous learning model In the The learning process for each domain is represented as follows: Cross-domain continuous learning models Perform parameter decomposition, using This refers to domain-specific parameters, which are parameters corresponding to the unique structures of different remote sensing domains. This is represented as a domain-shared parameter, which corresponds to the parameters shared by different remote sensing domains in the model structure; the model structure is an encoder-decoder structure, further expressed as: in, Decomposed into For encoder Parameters specific to the central domain These are domain-specific parameters in the corresponding decoder.

2. The remote sensing semantic segmentation method based on domain persistent learning according to claim 1, characterized in that: The Domain Residual Adaptive Module (DRA) structure. The Domain Residual Adaptive Module (DRA) reparameterizes domain-specific and domain-shared parameters through the model, extracting and training domain-shared and domain-specific features respectively. Specifically, the features extracted in the j-th module are: Features introduced by the domain residual adaptive module Connect the features passed in from the previous module j-1 in a hierarchical manner. composition, Among them, features Including domain shared portion Unique features of the domain The two features are combined, and the structure is as follows: in, It is a domain-shared structure, consisting of 3×1 and 1×3 convolutional layers, followed by the ReLU activation function; and It is a residual adaptive structure unique to parallel domains. It is a 1×1 convolutional layer. This indicates a batch normalization layer, where each domain has a unique domain-specific structure. and The features are combined in parallel between domains, that is, the features output by the previous module. Input to the domain shared structure respectively and the domain-specific structure corresponding to the current domain The combined features from both parts are then input into the domain-specific structure corresponding to the current domain. , to obtain features .

3. The remote sensing semantic segmentation method based on domain persistent learning according to claim 1, characterized in that: In step 3, the loss function that combines category filtering and feature space knowledge distillation consists of a category filtering distillation function, a feature space distillation function, and a cross-entropy function. (1) Distillation function based on category selection The cross-domain persistent learning model is trained sequentially on multiple domains. The model is then trained on the k-th domain. During training, it is represented as Current model Domain shared parameters Initialization based on past cross-domain continuous learning models Domain-shared parameters; The distillation process will separate the current domain data. Input into the cross-domain continuous learning model corresponding to the current domain Cross-domain continuous learning model corresponding to the past domain In this process, the predicted outputs of the cross-domain continuous learning model are distilled sequentially across multiple domains, using a distillation function based on category selection. for: in, and This represents the cross-domain continuous learning model corresponding to the current domain. Cross-domain continuous learning model corresponding to the past domain For each input The predicted output, This is expressed as the distillation loss function over the past domain, and sequentially over all past domains. Distillation is performed on the current domain; the distillation function based on category selection only performs distillation on the current domain. and past domain Knowledge distillation is performed in the common classes, with calculations performed in both the spatial and channel dimensions. in, Representing pixel-level binary labels, by... One-hot encoding is obtained. Indicates the current domain and past domain Common classes, Represents all pixels in the current domain; (2) Characteristic space distillation function Distillation function of characteristic space for: in, and This represents the cross-domain continuous learning model corresponding to the current domain. Cross-domain continuous learning model corresponding to the past domain encoder The output in the feature space, where Represented as encoder Corresponding input In the domain The output characteristics; This represents the L2 norm function, similar to the knowledge distillation loss function based on category selection, successively applied across all past domains. Distillation is carried out on top; (3) Cross-entropy loss function The cross-entropy loss function is: in, This represents the output of the current model. Represented as the cross-entropy function; (4) The total loss is a summary of all types of loss functions: in, They represent the cross-entropy function, respectively. Special category distillation function and characteristic space distillation function The weights; (5) Optimization strategy Cross-domain continuous learning model in the k-th domain During training, the optimization strategy for the cross-domain continuous learning model is as follows: for domain-specific parameters... Domain shared parameters Different optimization strategies are used in some areas. Parameter initialization is based on The decoder's classification layer is initialized randomly; furthermore, the current domain is frozen. Domain-specific parameters corresponding to other domains Domain shared parameters Initialization based on Corresponding shared parameters .

4. The remote sensing semantic segmentation method based on domain persistent learning according to claim 1, characterized in that: In step 4, the cross-domain continuous learning model is trained sequentially in multiple remote sensing domains using an incremental training and testing approach, as follows: Step 1: Based on the current domain k, New data and the cross-domain continuous learning model corresponding to the previous domain k-1 ; Step 2: Initialization, by adding new domain-specific layers. To model get ,based on In Parameter initialization In Simultaneously, freeze the domain-specific parameters except for the current domain. Step 3: Iterative training, for each batch of data in each iteration: Cross-domain continuous learning model corresponding to the current domain k Forward propagation through parameters ,calculate ; Backpropagation model Through parameters Backpropagation model Through parameters Calculate the distillation loss for both models. and ; Comprehensive calculation of losses ; Update the learning rate, including the domain-shared portion. and domain-specific parts Update using different learning rates; Cross-domain continuous learning model After training is completed in the current domain k, testing is performed on all past domains.

5. A remote sensing semantic segmentation system based on domain-based continuous learning, comprising: The system includes a data acquisition module, a data preprocessing module, a cross-domain continuous learning model module, and a training module. Data acquisition module: preprocesses the remote sensing domain sequence composed of multiple remote sensing domains to form a remote sensing domain sequence dataset; The preprocessing includes unified operations of data augmentation, image cropping, and label conversion; the dataset includes a training set and a test set. Data preprocessing module: Constructs a cross-domain continuous learning model as a semantic segmentation network; Cross-domain persistent learning model module: Based on the training set of the data acquisition module, the cross-domain persistent learning model in the data preprocessing module is trained sequentially in a certain remote sensing domain. A combination of category-based selection and feature space-based knowledge distillation is used as the loss function to mitigate catastrophic forgetting during the training process of the cross-domain persistent learning model, resulting in the trained cross-domain persistent learning model. Training module: The cross-domain continuous learning model is trained sequentially in multiple remote sensing domains using an incremental training and testing method. The training data only contains data from the current remote sensing domain. The model is tested on all previously accumulated remote sensing domains. After training all remote sensing domain sequences, the cross-domain continuous learning model is embedded into the remote sensing intelligent interpretation device to obtain the segmentation results of the multi-source remote sensing domain sequences. In the data preprocessing module, the structure of the cross-domain continuous learning model consists of a shared encoder. and K domain-specific decoders Composition; The shared encoder core structure is to seamlessly add multiple domain residual adaptive modules (DRA) to the ERFNet model. The structure of the ERFNet model is based on the improved ResNet, which replaces the standard 3×3 convolutional layer with a stack of 3×1 and 1×3 convolutional layers. The decoder structure consists of K parallel decoders, which is the same as the decoder structure in the ERFNet model. The number of decoders is the same as the number of remote sensing domains. When training and testing different remote sensing domains, the decoder corresponding to the domain is selected. For each remote sensing domain In the data These are remote sensing images and their corresponding semantic labels, used in the training of a cross-domain continuous learning model. These are the truth values ​​of the input and the corresponding output, respectively. Represented as predicted values; cross-domain continuous learning model In the The learning process for each domain is represented as follows: Cross-domain continuous learning models Perform parameter decomposition, using This refers to domain-specific parameters, which are parameters corresponding to the unique structures of different remote sensing domains. This is represented as a domain-shared parameter, which corresponds to the parameters shared by different remote sensing domains in the model structure; the model structure is an encoder-decoder structure, further expressed as: in, Decomposed into For encoder Parameters specific to the central domain These are domain-specific parameters in the corresponding decoder.

Images