Remote sensing image semantic change detection method based on multi-task twin network guided by enhanced change information

Abstract

The application discloses a remote sensing image semantic change detection method based on a multi-task twin network guided by enhanced change information, and comprises the following steps: a multi-task twin network based on enhanced change information guidance is constructed, the multi-task twin network comprises a change information enhancement module and a change information guidance module; the change information enhancement module is used for enhancing the feature expression of change information, and the change information guidance module is used for providing prior information of change positions for a semantic segmentation branch through a binary change detection branch; a loss function is constructed, and the multi-task twin network is trained in combination with the constructed loss function; the loss function comprises a semantic segmentation loss, a binary change loss and a semantic consistency loss; and remote sensing image semantic change detection is performed based on the trained multi-task twin network. The application further improves the performance of double-phase remote sensing image semantic change detection from three aspects of a fine multi-task learning structure design, change information enhancement and change information guidance.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image change detection technology, and in particular to a method for detecting semantic changes in remote sensing images based on a multi-task twin network guided by enhanced change information. Background Technology

[0002] Change detection (CD) of remote sensing images is an indispensable component of Earth observation missions. It is a technique for extracting information on changes in ground features by analyzing registered images of the same geographic area taken at different times. In recent decades, change detection has played an irreplaceable role in fields such as environmental monitoring, urban planning, disaster assessment, and military applications.

[0003] Most existing research focuses on change detection (CD) in bi-temporal RSIs. Based on the type of labels included in the output change map, CD can be divided into binary change detection (BCD) and semantic change detection (SCD). BCD outputs a change map with binary labels used to distinguish between changed and unchanged pixels in the two RSIs, such as... Figure 1 As shown in (a). Furthermore, SCD provides more detailed land cover category transformation information within the changed area, and its output change map has two representations, such as... Figure 1 As shown in (b), one approach is to provide "from-to" information using two semantic change maps containing information on the categories of changed areas. Another approach treats each "from-to" information as a change type, ultimately forming a multi-category change map; for example, five land cover categories can be combined into 21 change types (including invariant categories). Because the latter has a vast label space, most existing advanced SCD methods use two semantic change maps as the true labels for land cover. Compared to BCD, SCD provides richer land cover change information, meeting deeper research and application needs. However, SCD is a highly complex pixel-level interpretation task, facing numerous technical difficulties and challenges. It implicitly includes two subtasks: semantic segmentation (SS) and binary change detection (BCD). For the SS subtask, the limited number of changed areas in the image leads to insufficient available samples and sample imbalance, increasing the difficulty of semantic information extraction. For the BCD subtask, effectively utilizing dual-temporal information to accurately locate changed areas remains a persistent challenge in this field.

[0004] Traditional SCD methods primarily utilize manually crafted features or specific rules to predict land cover change types, such as image differencing, principal component analysis, change vector analysis, and Bayesian rules. These methods are highly sensitive to change thresholds. Furthermore, with advancements in remote sensing image acquisition technology, high-resolution remote sensing images have gradually become dominant in the data application market. Traditional SCD methods, due to their inherent poor robustness, cannot adaptively address the complex backgrounds and multi-scale features inherent in images. In recent years, the emergence of deep learning (DL) technology has driven the development of SCD; however, related research is still in its early exploratory stages. Relatively intuitive deep learning-based SCD methods include Comparison After Classification (PCC) and Direct Classification (DC). In these methods, the SS and BCD subtasks are intertwined, ignoring the complexity of the SCD task and the training difficulty of deep networks, thus limiting accuracy improvement. Recently emerging multi-task learning methods utilize twin encoders to extract bi-temporal deep features and model semantic and change information separately through multi-task decoding branches, becoming the mainstream approach for current SCD tasks.

[0005] Despite the progress made by existing SCD methods, several key challenges remain. First, remote sensing imagery, especially high-resolution imagery, provides highly detailed ground feature information, while existing SCD methods lack this refined representation, easily leading to missed or false detections. Second, for SCD tasks, existing methods for acquiring change features rely on relatively simple logical operations such as element-wise subtraction or feature concatenation, failing to adaptively mine change information and lacking effective change feature extraction methods. Finally, in existing multi-task learning methods, the BCD branch is used to identify changed regions, and the SS branch is used to determine the multi-temporal semantic types of changed regions, but the intrinsic relationship between the SS and BCD subtasks is not deeply explored. In conclusion, addressing these three challenges is crucial for improving the accuracy of SCD. Summary of the Invention

[0006] To address the three challenges mentioned above, this invention proposes a semantic change detection method for remote sensing images based on an Enhanced Change Information Guided Multi-Task Siamese Network (EGMS-Net). Specifically, EGMS-Net integrates refined semantic and change information through a coarse-to-fine "encode-decode" structure. Furthermore, a Change Information Enhancement Module (CIEM) is embedded in EGMS-Net to adaptively capture fine-grained change features, and a Change Information Guidance Module (CIGM) is embedded to strengthen the attention constraint of the change branch on the semantic branch. In summary, the proposed method improves the remote sensing image SCD method from three aspects: refined structure design, change information enhancement, and change information guidance.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for detecting semantic changes in remote sensing images based on a multi-task twin network guided by enhanced change information includes:

[0009] A multi-task twin network based on enhanced change information guidance is constructed. The multi-task twin network includes a change information enhancement module and a change information guidance module. The change information enhancement module is used to enhance the feature representation of change information, and the change information guidance module is used to provide prior information on the change position for the semantic segmentation branch through a binary change detection branch.

[0010] A loss function is constructed, and the multi-task Siamese network is trained using the constructed loss function; the loss function includes semantic segmentation loss, binary transformation loss, and semantic consistency loss;

[0011] Semantic change detection in remote sensing images based on a trained multi-task twin network.

[0012] Furthermore, the multi-task Siamese network also includes two Siamese encoders and a multi-task feature decoder. The encoders use ResNet34 as the backbone network and reshape the four-stage features of ResNet34 into multi-level semantic coding features. Corresponding to the two time phases before and after the change, E represents the encoder, and the two twin encoders use ResNet34 with shared weights. The multi-task feature decoder consists of two semantic segmentation decoders D. S1 D S2 A binary change detection decoder D C Composition, D S1 D S2 and D C Having the same network structure, and D S1 and D S2 Shared weights.

[0013] Furthermore, in D S1 Or D S2 In this context, the input consists of multi-level semantic features. The top-level feature X is obtained through three decoding stages. St The input features for the first stage are the top-level features of the encoder. The identity mapping is used, and each subsequent stage receives input from the multi-level features of the previous stage and the encoder output, and uses feature concatenation and 1×1 convolution operation to obtain the features of the next stage.

[0014] Furthermore, in the third stage of decoding, an upsampling operation is added before feature concatenation.

[0015] Furthermore, the change information enhancement module uses multi-level encoding features. As input, the enhanced multi-level variation characteristics of the output

[0016] Furthermore, in the change information enhancement module, for the dual-temporal coding features of the i-th level... First, the initial difference feature map D is obtained by subtracting elements one by one and taking the absolute value. i Subsequently, in D i The above calculation of attention yields A i A is multiplied by element-wise multiplication. i The refined two-phase features are obtained by passing them to the two-phase features. and And and Intermediate features are obtained by performing feature concatenation, and the number of channels is halved by using 1×1 convolution. Then The above calculation of attention is obtained Will and Performing element-wise multiplication yields A i With D i Performing element-wise multiplication yields Ultimately and The enhanced variation features are obtained by adding the features together.

[0017] Furthermore, in the change information enhancement module, the attention is calculated according to the following formula:

[0018]

[0019] Where X represents the feature, Att(·) denotes the computational spatial-spectral joint attention, and μ(X) and σ(X) represent the features. 2 Let X represent the mean and variance of X along the channel dimension, respectively, λ be a parameter, and Sigmoid(·) denote the Sigmoid activation function.

[0020] Furthermore, the D C The input is the enhanced multi-level change features from the change information enhancement module.

[0021] Furthermore, in the change information guidance module, D is processed in the following manner. S1 and D S2 Output feature X S1 and X C Processing:

[0022]

[0023]

[0024] Where Avgpool(·) and Maxpool(·) represent the average pooling and max pooling functions, respectively. This indicates a sequential concatenation of a 1×1 convolution, a ReLU function, and a 1×1 convolution; Sigmoid(·) represents the Sigmoid activation function. and Let represent element addition and element multiplication operations respectively, where t∈{1,2};

[0025] For D C Output feature X C Deep features F are extracted using the residual module. C The residual module consists of M residual blocks. Depthwise separable convolutions are used in each residual block, and F is calculated as follows: C Spatial attention (SA) C :

[0026] SA C =Sigmoid(Conv 7×7 (Conc(Avgpool(F C ),Maxpool(F C ))))

[0027] Where Conv 7×7 Represents a 7×7 convolution;

[0028] Then SA C respectively with F S1 F S2 F C Perform element-wise multiplication to obtain the output X′ of the multi-task branch. S1 、X′ S2 and X′ C .

[0029] Furthermore, the semantic segmentation loss employs a multi-class cross-entropy loss function between the prediction results of the two semantic segmentation branches and the true labels of semantically changed features; the binary change loss combines binary cross-entropy loss and Dice loss; the semantic consistency loss uses a cosine function to measure the similarity of probability distributions; the total loss function... Depend on and Common components:

[0030]

[0031] in and The semantic segmentation loss is represented by the sum of the two semantic segmentation branches and the average of the sum. This represents the loss of semantic consistency.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1) A change information enhancement module is proposed, which adaptively enhances change information from the multi-level coding features of dual-temporal images. In addition, a parameterless spatial-spectral joint attention mechanism (SSCA) is introduced into this module. Unlike commonly used two-dimensional attention mechanisms such as channel attention and spatial attention, SSCA is a three-dimensional attention mechanism that can jointly acquire spatial-spectral information and does not require additional parameters.

[0034] 2) A change information guidance module was designed to enhance the collaboration between the change branch and the semantic branch. It uses the spatial attention map of deep change features as a prior constraint to guide the semantic branch to perform semantic segmentation of the change region, which effectively alleviates the training difficulty of the network in the semantic branch.

[0035] 3) Based on the use of change information enhancement and change information guidance modules, a novel network, EGMS-Net, is proposed, which features a refined "encoder-decoder" structure design on the basis of existing multi-task learning methods. Experiments on two benchmark datasets show that the proposed EGMS-Net outperforms existing state-of-the-art SCD methods. Attached Figure Description

[0036] Figure 1 Diagram of BCD and SCD tasks;

[0037] Figure 2 A schematic diagram of the overall structure of EGMS-Net constructed according to an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the structure of the change information enhancement module constructed according to an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the structure of the change information guidance module constructed according to an embodiment of the present invention;

[0040] Figure 5 The semantic change detection results of the SECOND dataset provided in this embodiment of the invention;

[0041] Figure 6 This is a semantic change detection result diagram of the Landsat-SCD dataset provided in an embodiment of the present invention;

[0042] Figure 7The scores corresponding to different combinations of hyperparameters λ and N provided in the embodiments of the present invention;

[0043] Figure 8 The feature visualization results provided for embodiments of the present invention. Detailed Implementation

[0044] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0045] A method for detecting semantic changes in remote sensing images based on a multi-task twin network guided by enhanced change information includes:

[0046] A multi-task Siamese network EGMS-Net based on enhanced change information guidance is constructed. The EGMS-Net includes a change information enhancement module and a change information guidance module. The change information enhancement module is used to enhance the feature representation of change information, and the change information guidance module is used to provide prior information on the change location to the semantic segmentation branch through a binary change detection branch.

[0047] A loss function is constructed, and EGMS-Net is trained using the constructed loss function; the loss function includes semantic segmentation loss, binary transformation loss, and semantic consistency loss;

[0048] Semantic change detection in remote sensing images based on trained EGMS-Net.

[0049] (1) Overall network structure

[0050] The overall structure of the proposed EGMS-Net is as follows: Figure 2 As shown, given dual-temporal input images I1 and I2, multi-level semantic features are first extracted using two encoders E1 and E2 with shared weights. Furthermore, a change information enhancement module (CIEM) is employed to enhance the feature representation of change information, thereby obtaining corresponding multi-level change features. Where t∈{1,2} corresponds to the two time phases before and after the change, respectively. Subsequently, the multi-level semantic features and change features are input into the multi-task feature decoder, which mainly consists of two SS decoders with shared weights. S1 D S2 A BCD decoder D C Composition, through D S1 D S2 and D C Obtain the multi-task branch output X respectively S1 X S2 and X CSubsequently, a Change Information Guidance Module (CIGM) is embedded for subtask interaction, effectively focusing on semantic learning of changed regions in the image, and further predicting and generating three direct mappings: semantic mapping Y. S1 Y S2 Mapping Y with BCD C Finally, through Y C Mask Y respectively S1 and Y S2 Generate semantic change graphs Y1 and Y2.

[0051] (2) Refined multi-task twin network

[0052] To avoid parameter redundancy, EGMS-Net follows the popular multi-task learning architecture in the SCD field. It employs a siamese semantic encoder to extract multi-level semantic features from images and uses a multi-task feature decoder to recover the spatial information of the images layer by layer. Specifically, given dual-temporal images... Where H, W, and C represent the image's length, width, and number of spectral channels, respectively. To reduce the risk of overfitting, two ResNet34 modules with shared weights are embedded as the backbone network, forming the twin encoders E1 and E2. The four-stage features of ResNet34 are then reshaped into multi-level semantic coding features. and

[0053] The multi-task feature decoder consists of two SS decoders D S1 D S2 A BCD decoder D C Composition. D S1 D S2 and D C Having the same network structure, and D S1 and D S2 Shared weights. In D St In (t=1,2), the input is multi-level semantic features. The top-level feature X is obtained through three decoding stages. St Specifically, the input features for the first stage are the top-level features of the encoder. The identity mapping is used, and each subsequent stage receives input from the multi-level features of the previous stage and the encoder output, and uses feature concatenation and 1×1 convolution operations to obtain the features for the next stage. Specifically, in the third stage of decoding, due to the inconsistent spatial scales of the features to be concatenated, an upsampling operation is added before feature concatenation. St The formula for calculating (t=1,2) is as follows:

[0054]

[0055]

[0056]

[0057] Where i represents different decoding stages, Conc(·) represents the feature concatenation operation, up(·) represents the upsampling operation, and Conv 1×1 represents a 1×1 convolution, and BN and ReLU represent Batch Normalization (BN) and Rectified Linear Unit (ReLU), respectively. D C Network structure and computational process with D S1 D S2 The difference is that its input is the multi-level variation features enhanced by CIEM.

[0058] (3) Change Information Enhancement Module (CIEM)

[0059] Generally, change information is obtained directly through simple logical operations such as element-wise subtraction or feature concatenation. However, this method has poor adaptability to change detection tasks and the obtained change information contains a lot of noise, which seriously affects the accurate localization of the changed region. Therefore, a CIEM (Continuous Interpretation Matrix) is designed, such as... Figure 3 As shown, CIEM encodes features in multiple levels. The input and output are enhanced features. Using the bi-temporal coding features of the i-th level

[0060] For example, the initial difference feature map D is first obtained by subtracting element by element and taking the absolute value. i The calculation formula is as follows:

[0061]

[0062] Where |·| represents the absolute value operation. Subsequently, in D... i The above calculation of attention yields A i A is multiplied by element-wise multiplication. i The refined two-phase features are obtained by passing them to the two-phase features. and And and Intermediate features are obtained by performing feature concatenation, and the number of channels is halved by using 1×1 convolution. It should pass

[0063] The process can be represented as

[0064] A i =Att(D i (5)

[0065]

[0066]

[0067] Where Att(·) represents the calculation of SSCA, and These represent element addition and element multiplication operations, respectively.

[0068] Furthermore, in The above calculation of attention is obtained To refine and D i Feature representation, and Performing element-wise multiplication yields A i With D i Performing element-wise multiplication yields Ultimately and The enhanced variation features are obtained by adding the features together. Specifically, it is expressed as follows:

[0069]

[0070]

[0071]

[0072]

[0073] exist Figure 3 In visual processing, attention mechanisms play a crucial role in enhancing changing information. Most current research employs channel attention, spatial attention, and their variants to capture important information in feature maps, which introduces more parameters. Furthermore, these attention mechanisms focus solely on the channel or spatial dimension, lacking a collaborative representation of spatial and spectral information. Therefore, we introduce a parameter-free Spatial-Spectral Collaborative Attention (SSCA), a three-dimensional attention mechanism that collaboratively represents spatial and spectral information. SSCA is primarily designed based on the spatial inhibition phenomenon in visual neuroscience, where neurons with lower energy are more important in visual processing. The formula for calculating SSCA for feature X is as follows:

[0074]

[0075] Where μ(X) and σ(X) 2Let represent the mean and variance of X along the channel dimension, respectively. λ is a parameter, and Sigmoid(·) represents the Sigmoid activation function.

[0076] (4) Change Information Guidance Module (CIGM)

[0077] The SCD task only needs to identify changes in land cover categories in the changed area, while X S1 With X S2 The presence of both changing and non-changing regions increases the complexity of the SS subtask. To address this, a CIGM (Constant Increasing Generalization) mechanism was designed, which aims to provide prior information about the changing locations of the SS branches through BCD (Browser-Cardholder) branches. Figure 4 As shown in (a), the input feature of CIGM is X. S1 X S2 and X C Obtain X respectively S1 and X S2 Channel attention CA S1 and CA S2 The structure of channel attention is as follows Figure 4 As shown in (b). CA S1 CA S2 respectively with X S1 X S2 Element-wise multiplication yields F S1 and F S2 To achieve feature weighting along the channel dimension, the specific calculation formula is as follows:

[0078]

[0079]

[0080] Where Avgpool(·) and Maxpool(·) represent the average pooling and max pooling functions, respectively. This represents a sequential concatenation of a 1×1 convolution, a ReLU function, and a 1×1 convolution.

[0081] For the input X C The residual module is used to further extract deep features F. C The residual module consists of M residual blocks. For example... Figure 4 As shown in (c), to reduce the number of parameters in the residual module, depthwise separable convolutions are used in each residual block. Subsequently, based on... Figure 4 Structural calculation F of (d) C Spatial attention (SA) C , will SA C respectively with F S1 F S2 F CPerform element-wise multiplication to obtain the output X′ of the multi-task branch. S1 、X′ S2 and X′ C The specific formula is as follows:

[0082] SA C =Sigmoid(Conv 7×7 (Conc(Avgpool(F C ),Maxpool(F C (15)

[0083]

[0084]

[0085] Where Conv 7×7 This represents a 7×7 convolution. In CIGM, channel attention and residual modules are used to enrich feature representation. Based on this, spatial attention of the BCD branch is used as prior information for the variable region, which reduces the training burden of EGMS-Net on the SS branch and effectively explores the intrinsic relationship between the SS subtask and the BCD subtask.

[0086] Following CIGM, to obtain the final SCD prediction result, X′ was analyzed separately. S1 、X′ S2 and X′ C Perform convolution and upsampling operations to obtain the SS mapping Y of the dual-temporal images. S1 and Y S2 and BCD mapping Y C Then use Y C Mask Y S1 and Y S2 To generate Y1 and Y2.

[0087] (5) Loss Function

[0088] EGMS-Net is trained using three loss functions: semantic segmentation loss. Binary change loss and semantic consistency loss

[0089] The SS branch prediction result Y S1 Y S2 The multi-class cross-entropy loss function between semantically changed ground cover true labels L1 and L2 is calculated as follows:

[0090]

[0091] Where N is the number of semantic categories, and N = 0 represents "no-change". i and y i Let represent the true label and predicted probability of the i-th category of ground features, respectively. To ensure the network focuses on the SS task in changing regions, the "no-change" category is excluded. Beyond calculations.

[0092] Considering the imbalance between positive and negative samples in the BCD task, a binary cross-entropy loss is adopted. and Dice loss To jointly supervise the BCD branches. The calculation is as follows:

[0093]

[0094]

[0095]

[0096] Among them l c and y c Let l represent the predicted probabilities of actual land cover labels and changes in BCD, respectively. c It is obtained by replacing the non-zero label values ​​in L1 or L2 with 1.

[0097] Introducing semantic consistency loss specific to SCD task It uses the cosine function to measure the similarity of probability distributions, aiming to make the outputs of the two SS branches have similar probability distributions in the changing region and different probability distributions in the invariant region. The calculation formula is as follows:

[0098]

[0099] Where x1 and x2 represent Y respectively S1 With Y S2 The feature vector per pixel.

[0100] Total loss function Depend on and Together they form the following components, calculated as follows:

[0101]

[0102] in and The semantic segmentation losses of the two SS branches are represented by their sum and average.

[0103] To verify the effectiveness of the present invention, the following experiment was conducted:

[0104] 1. Experimental Setup

[0105] 1.1 Dataset Description

[0106] We conducted experiments on two publicly available datasets: the Semantic Change Detection (SECOND) dataset and the Landsat-SCD dataset [Yuan P, Zhao Q, Zhao X, et al. A transformer-based Siamesenetwork and an open optical dataset for semantic change detection of remote sensing images[J]. International Journal of Digital Earth, 2022, 15(1): 1506-1525.]. The former is a high-resolution dataset collected from many urban areas, while the latter is a medium-resolution dataset from the desert region of Northwest China. The differences in spatial resolution and observation scenes ensured data diversity and the reliability of the experiments.

[0107] The SECOND dataset is a commonly used and well-annotated high-resolution dataset in SCD (Sensitive Change Detection). It consists of 4662 pairs of remote sensing images acquired from different sensors and platforms, of which 2968 pairs are publicly available, covering multiple Chinese cities including Hangzhou, Chengdu, and Shanghai. Each image is 512×512 pixels in size, with ground sampling distances ranging from 0.5 meters to 3 meters. Each pair of data provides the original image and manually labeled data. The labels include one unchanged class and five land cover classes (no vegetation, trees, low vegetation, water, buildings, and playgrounds). From these six land cover classes, 30 "from-to" semantic change types can be derived. In the SECOND dataset, unchanged pixels account for more than 80% of the total pixels, while the 30 change types represent only a small proportion. This extreme class imbalance poses a significant challenge to the semantic change detection task. In our experiments, we divided the dataset into a training set (2079 pairs), a validation set (296 pairs), and a test set (593 pairs) in a 7:1:2 ratio.

[0108] The Landsat-SCD dataset was collected from Landsat images of the Tumushuk region of Xinjiang from 1990 to 2020. This region is adjacent to the Taklamakan Desert and has a fragile ecosystem. The dataset consists of 8468 image pairs. After removing images obtained through spatial augmentation operations (flipping, masking, scaling), the original data comprises 2385 pairs. Each image is 416×416 pixels in size, with a ground sampling distance of 30m. The dataset's annotations include one invariant class and four land cover classes: farmland, desert, buildings, and water. These four land cover classes can derive 10 "from-to" semantic change types. In the Landsat-SCD dataset, changed pixels account for only 18.89% of the total pixels, encompassing various complex detection scenarios. Furthermore, buildings exhibit a small and dispersed pattern, and the dual-temporal time span ranges from 3 to 23 years, better reflecting the characteristics of semantic change detection in real-world scenarios. In the experiment, we divided the dataset into a training set (1431 pairs), a validation set (477 pairs), and a test set (477 pairs) in a 3:1:1 ratio.

[0109] 1.2 Implementation Details

[0110] The experiments were conducted on a laptop equipped with an Intel Xeon Gold 5220 CPU and two Tesla V100 PCIe GPUs. The proposed EGMS-Net was primarily implemented using the PyTorch toolbox and the Python language. During model training, the batch size and number of iterations were set to 6 and 100, respectively. The Adamw optimizer was used to optimize the network weights, where weight decay and the initial learning rate (denoted as lr) were optimized. initial The values ​​are set to 0.0001 and 0.0003 respectively. The learning rate is calculated using the formula lr = lr initial ×(1-iterations / total_iterations) 0.9 The number of residual blocks is set to M = 6, and the parameter λ in formula (12) is set to 0.0001.

[0111] 1.3 Evaluation Indicators

[0112] This invention employs five commonly used evaluation metrics in semantic change detection to measure the detection accuracy of different methods: overall classification accuracy (OA), mean intersection-overall reunion ratio (mIoU), separation Kappa coefficient (SeK), and SCD-targeted F1 score (F...). scd ) and Score. To calculate the above metrics, we first need to define a confusion matrix Q = {q ij|0≤i≤N,0≤j≤N}, where N represents the number of categories, N=0 represents no-change, q ij OA represents the total number of pixels that actually belong to the j-th category and are predicted to be in the i-th category. OA can be defined as:

[0113]

[0114] OA (Adjustment Aspect Ratio) can only effectively characterize the model's accuracy when the number of positive and negative samples is roughly equal. In semantic change detection tasks, a large proportion of non-changing pixels will lead to an unreasonable OA. To mitigate the impact of label imbalance, two metrics, mIoU and Sek, are introduced. The formula for calculating mIoU is as follows:

[0115]

[0116]

[0117]

[0118] IoU1 measures the recognition accuracy of unchanged pixels, while IoU2 evaluates the extraction performance of changed regions. By combining the metrics of unchanged and changed pixels, mIoU, to some extent, focuses on changed regions in the image. Unlike mIoU, SeK focuses on evaluating semantic change detection in changed regions, based on a novel confusion matrix. The calculation yielded, where and The formula for calculating SeK is as follows:

[0119]

[0120]

[0121]

[0122] Introducing F scd The formula used to measure the accuracy of feature segmentation in a changed area is as follows:

[0123]

[0124]

[0125]

[0126] Where P scd and R scdThese represent precision and recall, respectively. Furthermore, a score index is calculated using mIoU and SeK weighting to comprehensively evaluate the benchmark method. The calculation formula is shown below:

[0127] Score = 0.3 * mIoU + 0.7 * SeK (34)

[0128] 2. Comparative Experiment

[0129] 2.1 Comparison Method

[0130] To verify the effectiveness of the proposed EGMS-Net in detecting semantic changes in remote sensing images, 12 state-of-the-art methods applicable to BCD and SCD tasks were selected for comparison. The specific information of the comparison methods is as follows.

[0131] FC-EF: Dual-temporal images are first fused in the early stage, and then change detection is performed through an encoder-decoder structure similar to UNet.

[0132] FC-Siam-Conc: A twin extension of FC-EF, it uses two encoders with shared weights to extract features from bi-temporal images and then gradually connects the features from the two encoders during the decoding process.

[0133] FC-Siam-diff: Unlike FC-Siam-Conc, FC-Siam-diff gradually integrates the differences between the twin encoders during the decoding stage.

[0134] IFN [Zhang C, Yue P, Tapete D, et al. A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 166: 183-200.]: It uses a twinned VGG16 as the encoder, introduces skip connections, channel attention and spatial attention mechanisms in the decoding stage, and adopts a deep supervision strategy during training.

[0135] HRSCD-str2 [Caye Daudt R, Le Saux B, Boulch A, et al. Multitask learning for large-scale semantic change detection[J]. Computer Vision and Image Understanding, 2019, 187: 102783.]: A direct classification method for SCD, which directly uses UNet as a classifier to obtain bi-temporal semantic change maps.

[0136] HRSCD-str3 [Caye Daudt R, Le Saux B, Boulch A, et al. Multitask learning for large-scale semantic change detection[J]. Computer Vision and Image Understanding, 2019, 187: 102783.]: A network designed for the SCD task, which separates the change detection task and the land cover mapping task. The network contains three encoders and three decoders.

[0137] HRSCD-str4 [Caye Daudt R, Le Saux B, Boulch A, et al. Multitask learning for large-scale semantic change detection[J]. Computer Vision and Image Understanding, 2019, 187: 102783.]: Based on HRSCD-str3, it connects the features of the encoder and decoder, and simultaneously passes the encoded information of the semantic mapping branch to the change detection branch through skip connections.

[0138] SCDNet: A twin UNet architecture for SCD, which introduces multi-scale dilated convolutions and attention mechanisms in the connection paths of the encoder and decoder, and employs a deep supervision strategy during training.

[0139] SSCD-1 [Ding L, Guo H, Liu S, et al. Bi-Temporal Semantic Reasoning for the Semantic Change Detection in HR Remote Sensing Images[J].IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-14.]: A popular and common three-branch structure in current SCD tasks, where semantic features are extracted separately through two temporal branches and then merged in the change encoder. Furthermore, a task-specific semantic consistency loss is introduced during training.

[0140] Bi-SRNet introduces two additional semantic reasoning modules on the basis of the SSCD-1 architecture, which utilize attention mechanisms to improve semantic representation and model semantic associations.

[0141] STSP-Net [He Y, Zhang H, Ning X, et al. Spatial-Temporal SemanticPerception Network for Remote Sensing Image Semantic Change Detection[J].Remote Sensing,2023,15(16):4095.]: In the encoding stage, spatial detail attention path and deep semantic extraction path are introduced respectively, and spatial semantic learning is enhanced by self-attention mechanism. A temporal refinement module is designed to enhance the sensitivity of temporal features to changing regions. In addition, a concise invariant consistency loss function is developed based on metric learning strategy.

[0142] MTSCD-Net [Cui F, Jiang J. MTSCD-Net: A network based on multi-task learning for semantic change detection of bitemporal remote sensing images[J]. International Journal of Applied Earth Observation and Geoinformation, 2023, 118: 103294.]: A twin semantic encoder was constructed based on the Swin-Transformer, and dual-temporal features were fused through an aggregation module. Subsequently, the accuracy of SCD was effectively improved through a change information extraction module and a spatial feature enhancement module.

[0143] Of the 12 comparison methods mentioned above, FC-EF, FC-Siam-Conc, FC-Siam-diff, and IFN were originally designed for BCD, while the remaining 8 are state-of-the-art methods specific to the SCD task. To apply the four BCD methods to the SCD task, we modified the number of convolutional layers and channels at the top of the network to output two semantic change prediction maps.

[0144] 2.2 Results on the SECOND dataset

[0145] Table 1 lists the quantitative results of different methods on the SECOND dataset, from which three aspects of the analysis can be observed. First, the SCD method achieved higher evaluation metrics than the BCD method, indicating that the SCD task is more complex than the BCD task, and directly applying BCD-related networks to the SCD task cannot achieve ideal experimental results. Second, among SCD-related methods, multi-task learning methods (HRSCD-str4, SSCD-l, Bi-SRNet, STSP-Net, MTSCD-Net, EGMS-Net) achieved higher values ​​on most evaluation metrics. Multi-task learning methods decompose the SCD task into SS sub-tasks and BCD sub-tasks, reducing the complexity of the task and the training difficulty of the network. Compared to HRSCD-str4, SSCD-l, Bi-SRNet, STSP-Net, MTSCD-Net, and EGMS-Net further consider the interaction of bi-temporal features and the correlation between sub-tasks, thus achieving higher scores on SeK and F. scd The proposed EGMS-Net achieves numerical leadership across various evaluation metrics. Third, it outperforms the comparison methods on all evaluation metrics, specifically with an OA of 87.40%, equal to that of SCD-Net. (mIOU, SeK, F) scd The scores are 0.46%, 0.72%, 0.51%, and 0.65% higher than the suboptimal method, respectively, which demonstrates the superior performance of EGMS-Net in the SCD task.

[0146] Table 1. Quantitative comparison of different methods on the SECOND dataset for OA, mIOU, Sek, Fscd, and Score. Bold values ​​represent the best values ​​in the corresponding columns.

[0147]

[0148] In addition to quantitative comparisons of different methods, Figure 5 The document also shows visualizations of different methods on the SECOND dataset. Figure 5In the image, (a1) and (a2), and (b1) and (b2) represent different image pairs and their SCD results, respectively. Referring to the actual land cover labels, it can be observed that the BCD correlation method suffers from severe false negatives. Among the three HRSCD series methods, HRSCD-str4 yielded better quality SCD results. In contrast, the semantic change maps of the four methods SSCD-l, Bi-SRNet, STSP-Net, and MTSCD-Net have fewer false positives, but the completeness of the change regions and the discriminative power of similar categories are still insufficient. In particular, the proposed EGMS-Net can obtain more satisfactory SCD results, and its superiority is mainly reflected in three aspects. First, it captures richer land cover details, which is mainly due to the refined encoder-decoder structure. For example, in... Figure 5 In the yellow-framed regions (b1) and (b2), EGMS-Net captured more detailed information about the changes "from tree to ground". Secondly, because CIEM enhances the expressive power of change features, the spatial structure of the change regions is more complete; for details, please refer to... Figure 5 The purple-framed areas in (a1) and (a2). Third, as... Figure 5 As shown in the purple boxes in (b1) and (b2), EGMS-Net is more discriminative of class confusion between Tree and Low vegetation. This is mainly due to CIGM effectively reducing the training difficulty of the SS branch, making the class prediction of the changing region more accurate.

[0149] 2.3 Results on the Landsat-SCD dataset

[0150] Table 2 lists the quantitative results of different methods on the Landsat-SCD dataset. The conclusions observed and analyzed from these results are basically consistent with those in Table 1. The performance of BCD-related methods and HRSCD series methods is relatively poor. Among the remaining comparative methods, the accuracy improvements of SSCD-1, Bi-SRNet, and STSP-Net are not significant. This is because these three methods only use a classification head to obtain SCD results after the Siamese decoder, lacking a multi-task decoder process for accurately representing spatial information. Conversely, SCDNet and MTSCD-Net design a multi-stage feature decoding process and embed channel attention and spatial attention into the network structure, thus effectively improving SCD performance. The proposed method EGMS-Net performs best in the quantitative metrics of SCD. Specifically, compared to the second-best method MTSCD-Net, EGMS-Net improves OA, mIOU, SeK, Fscd, and Score by 0.96%, 2.26%, 7.30%, 2.55%, and 5.79%, respectively. The SeK value is significantly improved.

[0151] Figure 6 This presentation visually demonstrates the SCD results of different methods on the Landsat-SCD dataset. Observing the example images and corresponding ground truth markers reveals that the changed areas are characterized by continuous linear features (e.g., water) and dense block features (e.g., buildings, farmlands, and deserts). BCD-related methods and the HRSCD series of methods exhibit significant false negatives and are unable to effectively identify these features. Figure 6 The disappearance of Water in (a1) and (a2) and Figure 6 The addition of Buildings in (b1) and (b2) shows that the SSCD-1, Bi-SRNet, and STSP-Net methods cannot accurately identify the Water category and have weak connectivity between pixels. In contrast, the SCDNet, MTSCD-Net, and EGMS-Net methods yield more coherent land cover distributions, and the visualization results are closer to the actual land cover labels. These phenomena are consistent with the quantitative results in Table 2. Since Landsat-SCD is a medium-resolution remote sensing image, the land cover outlines are relatively blurred, so the visualization results of SCDNet, MTSCD-Net, and EGMS-Net are similar. However, EGMS-Net has higher accuracy, indicating less noise. Overall, for medium-resolution remote sensing images, EGMS-Net's qualitative and quantitative SCD results are better than other methods.

[0152] Table 2 shows a quantitative comparison of different methods on the Landsat-SCD dataset for OA, mIOU, Sek, Fscd, and Score. The bolded values ​​represent the best values ​​in their respective columns.

[0153]

[0154] 2.4 Model Efficiency Comparison

[0155] We used model parameters (params) and floating-point operations (FLOPs) to measure the parameter count and overall complexity of EGMS-Net and its comparative methods, where FLOPs were obtained when the image batch size was 1. Detailed results are shown in Table 3. The FC-EF, FC-Siam-Conc, FC-Siam-diff, and HRSCD series methods have lower parameter counts and complexity, but their simple network structures also prevent them from achieving satisfactory SCD results. Compared to MTSCD-Net, which has the closest evaluation metrics among the comparative methods, EGMS-Net requires fewer parameters and lower computational cost, indicating that the proposed method can balance detection accuracy and model efficiency.

[0156] Table 3 compares the efficiency of different methods on the SECOND and Landsat-SCD datasets.

[0157]

[0158]

[0159] 3. Discussion and Analysis

[0160] 3.1 Hyperparameter Analysis

[0161] The hyperparameters that need to be determined in EGMS-Net are mainly two: Figure 4 In section (c), the number M of residual blocks embedded in the residual module and the parameter λ in SSAC are used. To analyze the influence of different hyperparameters on the SCD results, a grid search strategy is used to determine the combination of values ​​for λ and M. Specifically, λ is taken from the set {0.00001, 0.00005, 0.0001, 0.0005, 0.0001}, and M is taken from the set {0, 2, 4, 6, 8}, where M = 0 indicates the removal of residual modules. Figure 7 This paper demonstrates the impact of different combinations of hyperparameters λ and M on the score on two datasets, where (a) is the SECOND dataset and (b) is the Landsat-SCD dataset. It can be observed that when λ is fixed, M=0 has the lowest score value, indicating that the residual blocks are helpful in extracting deep features. As the number of residual blocks M increases, the score generally shows an upward trend. The score values ​​for M=6 and M=8 are close; to avoid model parameter redundancy, the value of M is set to 6. When M is fixed, setting λ to 0.0001 can effectively improve the score value. Therefore, for both the SECOND and Landsat-SCD datasets, the hyperparameters are uniformly set to M=6 and λ=0.0001.

[0162] 3.2 Ablation Study

[0163] To evaluate the auxiliary components in EGMS-Net, further ablation studies were conducted on the SECOND and Landsat-SCD datasets, and the quantitative results are shown in Table 4. First, using a refined multi-task "encoder-decoder" structure as a baseline, it can be seen that the baseline method has relatively poor SCD accuracy, indicating that it is difficult to handle complex tasks in SCD using only a multi-task learning architecture. Subsequently, the CIEM and CIGM modules were added to the baseline, resulting in significant improvements in SCD metrics on both datasets. Taking the Landsat-SCD dataset as an example, embedding CIEM improved OA, mIOU, SeK, Fscd, and Score by 0.51, 1.41, 4.14, 1.31, and 3.31, respectively, while embedding CIGM improved OA, mIOU, SeK, Fscd, and Score by 0.83, 2.09, 6.76, 2.35, and 5.36, respectively, directly demonstrating that CIEM and CIGM played a crucial role in improving SCD performance. Finally, CIEM and CIGM are simultaneously embedded in the baseline to form the proposed EGMS-Net. CIEM optimizes the localization of changed regions, while CIGM excites the network to classify land cover categories within these regions. The effective combination of the two further improves detection accuracy, demonstrating the effectiveness of CIEM and CIGM in jointly addressing the SCD task.

[0164] Table 4 Ablation experimental results on the SECOND and Landsat-SCD datasets.

[0165]

[0166] 3.3 Advantages of CIEM

[0167] To verify the superiority of CIEM for the SCD task, it was compared with different variation feature extraction methods, and the results are shown in Table 5. Here, Sub, Sum, and Conc represent three simple logical operations: feature subtraction, feature addition, and feature concatenation, respectively. DEM and TFIM are two recently proposed variation feature extraction modules. DEM first uses feature subtraction and absolute value operations to obtain initial variation features, and then uses channel attention of the initial variation features to correct the bi-temporal features. Subsequently, it uses the corrected features to obtain the variation features. TFIM combines the three operations Sub, Sum, and Conc using 3×3 convolutions, effectively enhancing the interaction between bi-temporal features. Regarding model parameters, Sub, Sum, and Conc have no parameter additions, while DEM and CIEM have a small number of parameter additions, which come from channel attention in DEM and 1×1 convolutions and ReLU operations in CIEM, respectively. TFIM uses a large number of 3×3 convolutions, thus significantly increasing the number of parameters. Regarding accuracy metrics, the comparative methods cannot simultaneously handle both the high-resolution SECOND dataset and the medium-resolution Landsat-SCD dataset. For example, TFIM achieves high accuracy on the SECOND dataset, but its results on the Landsat-SCD dataset are lower than those of the Conc method. In particular, CIEM achieves the best results on both the SECOND and Landsat-SCD datasets, indicating that the proposed CIEM is more robust to different datasets.

[0168] Table 5 Results of CIEM Module Effectiveness Analysis

[0169]

[0170]

[0171] 3.4 CIGM Visualization Analysis

[0172] To intuitively analyze how CIGM improves SCD performance, we will... Figure 4 Some features in the image were visualized, such as Figure 8 As shown. Rows 1 and 2 correspond to the SECOND dataset, rows 3 and 4 correspond to the Landsat-SCD dataset, and T1 and T2 represent the two temporal phases before and after the change, respectively. Columns 1 to 6 represent the dual-temporal image and the DIGM input feature X, respectively. S1 With X S2 Grad-CAM visualization, spatial attention SA C Grad-CAM visualization, DIGM output feature X′ S1 With X′ S2Grad-CAM visualization, prediction results, and ground feature labels. In the Grad-CAM visualization map, red represents areas with high attention, and blue represents areas with low attention. It can be observed that before the features of the SS branch flow into CIGM, X... S1 With X S2 The primary focus is on scattered clusters of small buildings and mountainous terrain in the imagery, as shown in the area enclosed by the yellow dashed box in the figure. The spatial attention (SA) of the BCD branch... C The focus is mainly on areas of change, as shown in the area enclosed by the red dashed box in the figure. After spatial attention guidance via the BCD branch, the output feature X′ is obtained. S1 With X′ S2 Attention to the region within the yellow dashed box decreases, while attention to the region within the red dashed box relatively increases. Overall, guided by the spatial location of the BCD branch, the SS branch focuses more on the changing regions, which reduces the complexity of the SS subtask to some extent, thus yielding more refined segmentation results.

[0173] In summary, this invention proposes a novel EGMS-Net, which further enhances the performance of semantic change detection in bi-temporal remote sensing images through three aspects: refined multi-task learning structure design, change information enhancement, and change information guidance. A coarse-to-fine twin encoder and multi-task decoder are integrated to extract bi-temporal and change features layer by layer. CIEM, combined with parameter-free spatial-spectral joint attention, achieves adaptive enhancement of change information. In CIGM, the change branch provides spatial location guidance for the semantic branch, strengthening the interaction between temporal and change features and reducing the training difficulty of the network. Experimental results on the SECOND and Landsat-SCD datasets validate the effectiveness of the proposed EGMS-Net.

[0174] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting semantic changes in remote sensing images based on a multi-task twin network guided by enhanced change information, characterized in that, include: A multi-task twin network guided by enhanced change information is constructed, wherein the multi-task twin network includes a change information enhancement module and a change information guidance module; The change information enhancement module is used to enhance the feature representation of change information, and the change information guidance module is used to provide prior information on the change location to the semantic segmentation branch through the binary change detection branch; A loss function is constructed, and the multi-task Siamese network is trained using the constructed loss function; the loss function includes semantic segmentation loss, binary transformation loss, and semantic consistency loss; Semantic change detection in remote sensing images based on a trained multi-task twin network; The multi-task Siamese network also includes two Siamese encoders and a multi-task feature decoder. The encoders use ResNet34 as the backbone network and reshape the four-stage features of ResNet34 into multi-level semantic coding features. , Corresponding to the two time phases before and after the change, E represents the encoder, and the two twin encoders use ResNet34 with shared weights. The multi-task feature decoder consists of two semantic segmentation decoders. , A binary change detection decoder composition, , and Having the same network structure, and and Shared weights; In the aforementioned change information enhancement module, for the first... Two-phase coding features at each level , First, the initial difference feature map is obtained by subtracting elements one by one and taking the absolute value. Subsequently, The above calculation of attention is obtained Through element-wise multiplication The refined two-phase features are obtained by passing them to the two-phase features. and and will and Intermediate features are obtained by performing feature concatenation, and then... Convolution halves the number of channels, resulting in Then The above calculation of attention is obtained ,Will and Performing element-wise multiplication yields ,Will and Performing element-wise multiplication yields Ultimately and The enhanced variation features are obtained by adding the features together. ; In the change information enhancement module, attention is calculated according to the following formula: in Indicates features, This represents computational spatial-spectral joint attention. and They represent Mean and variance along the channel dimension, For parameters, This represents the Sigmoid activation function; In the change information guidance module, the following methods are used to... and Output characteristics and Processing: in and These represent the average pooling and max pooling functions, respectively. express Convolution, ReLU function and Sequential concatenation, This represents the Sigmoid activation function. and These represent element-wise addition and element-wise multiplication operations, respectively. ; for Output characteristics Deep features are extracted using residual modules. The residual module is composed of It consists of residual blocks, and depthwise separable convolution is used in each residual block, calculated as follows: Spatial attention : in express convolution; Then respectively with , , Perform element-wise multiplication to obtain the output of the multi-task branch. , and .

2. The method for detecting semantic changes in remote sensing images based on a multi-task twin network guided by enhanced change information as described in claim 1, characterized in that, exist or In this context, the input consists of multi-level semantic features. The top-level features are obtained through three decoding stages. The input features for the first stage are the top-level features of the encoder. The identity mapping is used, and each subsequent stage receives input from the multi-level features of the previous stage and the encoder output, and uses feature concatenation and... Convolution operations are used to obtain features for the next stage.

3. The method for detecting semantic changes in remote sensing images based on a multi-task twin network guided by enhanced change information as described in claim 2, characterized in that, In the third stage of decoding, an upsampling operation is added before feature concatenation.

4. The method for detecting semantic changes in remote sensing images based on a multi-task twin network guided by enhanced change information as described in claim 1, characterized in that, The change information enhancement module uses multi-level coding features. As input, the enhanced multi-level variation characteristics of the output .

5. The method for detecting semantic changes in remote sensing images based on a multi-task twin network guided by enhanced change information as described in claim 4, characterized in that, The The input is the enhanced multi-level change features from the change information enhancement module. .

6. The method for detecting semantic changes in remote sensing images based on a multi-task twin network guided by enhanced change information as described in claim 1, characterized in that, The semantic segmentation loss employs a multi-class cross-entropy loss function between the prediction results of the two semantic segmentation branches and the true labels of semantically changed features; the binary change loss combines binary cross-entropy loss and Dice loss; the semantic consistency loss uses a cosine function to measure the similarity of probability distributions; the total loss function... Depend on , and Common components: in and The semantic segmentation loss is represented by the sum of the two semantic segmentation branches and the average of the sum. , This represents the loss of semantic consistency.

Images