A remote sensing image change detection method and system based on semantic fusion

Abstract

The application provides a remote sensing image change detection method and system based on semantic fusion, and belongs to the technical field of data processing and application of remote sensing images. The method comprises the following steps: inputting multiple batches of first to-be-detected images and second to-be-detected images into a deep encoder with a local interactive enhancement module to extract local features; inputting first local feature maps and second local feature maps into a visual self-attention encoder to extract global features; inputting the global feature maps into a cascaded upsampling decoder, performing multiple upsampling and feature fusion operations, and obtaining feature mapping; inputting the feature mapping into a classifier for classification to obtain a remote sensing image change detection map. The effective fusion of local and global semantic features is realized, the sensitivity and accuracy of change detection of various ground objects are enhanced, and the change area is accurately positioned.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image data processing and application technology, specifically relating to a method and system for detecting changes in remote sensing images based on semantic fusion. Background Technology

[0002] Change detection (CD) in remote sensing imagery refers to the process of detecting surface changes using georeferenced multi-temporal remote sensing images. As a crucial technique for rapidly and effectively acquiring remotely sensed changes, image CD has attracted widespread attention in many remote sensing-related tasks, such as land use surveys, urban planning, environmental monitoring, and disaster mapping. Because high-resolution remote sensing images can provide rich spatial information and fine surface details, detecting changes in them is of great significance for fine-grained applications such as precision agriculture and urban renewal. However, CD in high-resolution remote sensing images faces many challenges, such as spurious changes, scale variations, and class imbalances. Traditional methods mostly rely on manual feature extraction, such as Principal Component Analysis (PCA), Gabor filtering, Multivariate alteration detection (MAD), and Change Vector Analysis (CVA). These methods can achieve CD tasks to a certain extent, but they have the following two drawbacks: First, the image features extracted by traditional methods are easily affected by seasonal changes, lighting conditions and satellite sensors, making it difficult to achieve high change detection accuracy; second, the manually extracted features rely heavily on prior knowledge of the domain, which limits the generalization ability of CD methods.

[0003] The rise of deep learning technology has led to the proposal of many new image processing (CD) methods. Among them, Convolutional Neural Networks (CNNs) have strong discriminative capabilities and have been successfully applied to CD tasks. CNN-based methods convert bi-temporal images into high-level spatial depth features and extract highly abstract contextual semantics of change regions, minimizing error propagation and reducing human error introduced by preprocessing. However, since CNNs can only extract global information by gradually expanding their receptive field, current CNN-based CD methods cannot fully realize spatial contextual information modeling. Recent research on remote sensing image CD shows that the Transformer model (a deep neural network model based on a self-attention mechanism) breaks through the limitations of the local receptive field of CNNs and can provide long-distance dependencies for semantic changes. The Transformer model has achieved excellent results in computer vision applications such as image classification, object detection, and semantic segmentation. However, using only the Transformer as an encoder ignores the importance of local information and increases computational complexity. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application proposes a method and system for remote sensing image change detection based on semantic fusion. It integrates a Transformer model and a convolutional neural network, employing a skip connection mechanism in the decoder to cascade multi-scale local detail features with global contextual features, achieving effective fusion of local and global semantic features. The proposed fusion method can enhance the sensitivity and accuracy of detecting changes in various land features while maintaining precise location of changed areas, thereby improving the method's adaptability and robustness in various scenarios.

[0005] Firstly, this application proposes a method for detecting changes in remote sensing images based on semantic fusion, including:

[0006] Multiple batches of first and second images to be detected are acquired, wherein the first and second images to be detected are dual-temporal images of the same region.

[0007] The first image to be detected and the second image to be detected are respectively input into a depth encoder with a local interaction enhancement module to extract local features and obtain a first local feature map and a second local feature map respectively.

[0008] The first local feature map and the second local feature map are respectively input into the visual self-attention encoder to extract global features and obtain a global feature map;

[0009] The global feature map is input into a cascaded upsampling decoder, and after multiple upsampling and feature fusion operations, a feature map is obtained.

[0010] The feature map is input into a classifier for classification to obtain a remote sensing image change detection map.

[0011] The deep encoder with local interaction enhancement modules includes a first layer, a second layer, a third layer, and a fourth layer. Local interaction enhancement modules are set between the second and third layers, between the third and fourth layers, and after the fourth layer. The first layer includes a convolutional layer and a pooling layer. The second layer includes a first residual module and three second residual modules. The third layer includes a first residual module and four second residual modules. The fourth layer includes a first residual module and nine second residual modules.

[0012] The local interaction enhancement module includes a spatial interaction layer and a channel interaction layer. A spatial interaction layer is set between the second and third levels, a channel interaction layer is set between the third and fourth levels, and a spatial interaction layer is set after the fourth level. The spatial interaction layer is used to perform feature exchange in the spatial dimension, and the channel interaction layer is used to perform feature exchange in the channel dimension.

[0013] The spatial interaction layer or channel interaction layer is calculated as follows:

[0014]

[0015] Where x0 and x1 represent the dual-phase characteristics, x 0 / 1 () represents the spatial interaction layer, n represents the batch size, c represents the number of channels for the first or second image to be detected, h represents the height of the resolution, w represents the width of the resolution, and M represents the exchange mask, which consists of 1 and 0, where 1 indicates exchange and 0 indicates no exchange.

[0016] The step of inputting the first local feature map and the second local feature map into the visual self-attention encoder respectively to extract global features and obtain a global feature map includes:

[0017] The first local feature map and the second local feature map are concatenated to obtain a concatenated feature map.

[0018] The cascaded feature map is unfolded into a two-dimensional image block, and the two-dimensional image block is linearly projected into a D-dimensional embedding space;

[0019] The positional information of the projected two-dimensional image block is encoded to obtain the encoded features;

[0020] The encoded features are input into a visual self-attention encoder to obtain a global feature map. The visual self-attention encoder includes a multi-head self-attention layer and a multi-layer perceptron layer. The multi-head self-attention layer calculates the attention value of each head and concatenates the attention values ​​of all heads. The concatenated result is used as the output of the multi-head self-attention layer. The attention value of each head is calculated by applying an attention mechanism to the output of each Transformer layer. Each layer of the multi-layer perceptron layer contains a set of non-linear activation functions.

[0021] The calculation formulas for the multi-head self-attention layer and the multi-layer perceptron layer are as follows:

[0022] Z′ l =MSA(LN(Z) l-1 ))+Z l-1

[0023] Z l =MLP(LN(Z′) l ))+Z′ l

[0024] Among them, Z l-1 Z′ is the encoded feature output by the (l-1)th Transformer layer. l Z represents the features output after computation by a multi-head self-attention layer. l Let L be the encoded feature output by the l-th Transformer layer after multi-layer perceptron computation, where l∈{1,…,L}, L is the total number of layers in the visual self-attention encoder, and LN(·) is the layer normalization operation, calculated as follows: Where x is the vector representation of the neurons in the l-th layer, g is the scaling parameter, b is the translation parameter, f is the activation parameter, μ is the mean of all neurons in the l-th layer on the training sample, and σ is the variance of all neurons in the l-th layer on the training sample.

[0025] The multi-head self-attention layer calculates the attention value for each head separately, and concatenates the attention values ​​of all heads. The concatenated result is used as the output of the multi-head self-attention layer. The attention value of each head is calculated by applying an attention mechanism to the output of each Transformer layer, as shown in the following formula:

[0026] MSA(Q,K,V)=Concat(head1,head2,…,head n W 0

[0027] head i =Attention(Z) l-1 W i Q Zl-1 W i K Z l-1 W i V )

[0028] Among them, Z l-1 The encoded features are the output of the (l-1)th Transformer layer. For querying the matrix, The key matrix, For value matrices, The parameter matrix of the multi-head self-attention layer, Concat() represents the cascade operation, head i Let be the attention value of the i-th self-attention head, e be the number of self-attention heads, and d be the attention value of the i-th self-attention head. k =d v =D / e, where D is the dimension of the D-dimensional embedding space.

[0029] The step of inputting the global feature map into a cascaded upsampling decoder, performing multiple upsampling and feature fusion operations to obtain a feature map includes:

[0030] The global feature map is input into multiple upsampling layers for upsampling processing to obtain multiple upsampling features. Each upsampling layer includes: one 2x upsampling operator block and two convolutional blocks. Each convolutional block includes: a 3×3 convolutional layer, a first batch of normalization layers, and a first ReLU activation function. The upsampling operator block uses bilinear interpolation to perform upsampling.

[0031] By using skip connections, each multi-scale feature of the global feature map is connected to each upsampled feature to obtain the feature map.

[0032] The classifier includes: two 1×1 convolutional layers, a second ReLU activation function, and a second batch normalization layer.

[0033] Secondly, this application proposes a remote sensing image change detection system based on semantic fusion, comprising:

[0034] The image acquisition module is used to acquire a first image to be detected and a second image to be detected in multiple batches. The first image to be detected and the second image to be detected are dual-temporal images of the same region.

[0035] The local feature extraction module is used to input the first image to be detected and the second image to be detected into the depth encoder with the local interaction enhancement module, respectively, to extract local features and obtain the first local feature map and the second local feature map.

[0036] The global feature extraction module is used to input the first local feature map and the second local feature map into the visual self-attention encoder respectively to extract global features and obtain a global feature map.

[0037] The upsampling decoding module is used to input the global feature map into the cascaded upsampling decoder, perform multiple upsampling and feature fusion operations to obtain the feature map;

[0038] The image classification module is used to input the feature map into the classifier for classification, and obtain the remote sensing image change detection map.

[0039] Beneficial effects:

[0040] This application proposes a method and system for detecting changes in remote sensing images based on semantic fusion. First, a deep encoder with a local interaction enhancement module encodes pixel blocks of dual-temporal remote sensing images. Then, a visual self-attention encoder is introduced to further mine the change features extracted by the deep encoder, fully exploring and integrating the global contextual information of the image. Finally, in the decoding stage, upsampling techniques are used to gradually restore the encoded high-level semantic features to the original spatial resolution of the image, achieving effective fusion of local and global semantic features. This allows for accurate identification of changed regions with more complete boundaries. Attached Figure Description

[0041] Figure 1 A flowchart of a remote sensing image change detection method based on semantic fusion according to an embodiment of this application;

[0042] Figure 2 A schematic diagram of the remote sensing image change detection method based on semantic fusion in this application embodiment;

[0043] Figure 3 A schematic diagram of the deep encoder structure with a local interaction enhancement module according to an embodiment of this application;

[0044] Figure 4 This application contains a schematic diagram of channel interaction in an embodiment.

[0045] Figure 5 A schematic diagram of the visual self-attention encoder structure according to an embodiment of this application;

[0046] Figure 6 A schematic diagram of the cascaded upsampling decoder structure in this application embodiment is shown.

[0047] Figure 7 A schematic diagram illustrating the visualization analysis results of an embodiment of this application;

[0048] Figure 8 A flowchart illustrating the visualization analysis results of this application embodiment;

[0049] Figure 9 A schematic diagram of a remote sensing image change detection system based on semantic fusion according to an embodiment of this application. Detailed Implementation

[0050] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0051] The ability to extract both local and global features is crucial in digital change detection (CD) methods and has received widespread attention in recent research. CNNs have become the mainstream framework for extracting local features from remote sensing images in many advanced CD methods. Transformer models offer an effective method for extracting global features from remote sensing images. This application proposes a remote sensing image change detection method and system based on semantic fusion. The core components of this method include a Real-Time Executive (REX) module, a Vision Transformer (VIT) self-attention encoder, and a Cascaded Upsampling Decoder (CUD). In the encoding stage, the two-temporal images are processed by the weight-sharing Real-Time Executive module to extract multi-scale local semantic features and achieve spatial and channel-level feature information interaction. Then, the REX output features are concatenated with the two-temporal semantic features and input into the VIT to extract global semantic features, achieving the fusion of local and global features from the two-temporal images. In the decoding stage, a decoder CUD containing multiple upsampling blocks is designed to decode hidden features, gradually restoring the features to the original image scale.

[0052] Example 1:

[0053] This embodiment proposes a remote sensing image change detection method based on semantic fusion, such as... Figure 1 , Figure 2 As shown, it includes:

[0054] Step S1: Obtain the first image to be detected and the second image to be detected, wherein the first image to be detected and the second image to be detected are dual-temporal images of the same region;

[0055] Step S2: Input the first image to be detected and the second image to be detected into the depth encoder with the local interaction enhancement module respectively, extract local features, and obtain the first local feature map and the second local feature map respectively;

[0056] Step S3: Input the first local feature map and the second local feature map into the visual self-attention encoder respectively to extract global features and obtain a global feature map;

[0057] Step S4: Input the global feature map into the cascaded upsampling decoder. After multiple upsampling and feature fusion operations, the feature map is obtained.

[0058] Step S5: Input the feature map into the classifier for classification to obtain the remote sensing image change detection map.

[0059] In step 2, the deep encoder with local interaction enhancement modules includes: a first layer, a second layer, a third layer, and a fourth layer. Local interaction enhancement modules are set between the second and third layers, between the third and fourth layers, and after the fourth layer. The first layer includes: a convolutional layer and a pooling layer; the second layer includes: a first residual module and three second residual modules; the third layer includes: a first residual module and four second residual modules; and the fourth layer includes: a first residual module and nine second residual modules.

[0060] The local interaction enhancement module includes a spatial interaction layer and a channel interaction layer. A spatial interaction layer is set between the second and third levels, a channel interaction layer is set between the third and fourth levels, and a spatial interaction layer is set after the fourth level. The spatial interaction layer is used to perform feature exchange in the spatial dimension, and the channel interaction layer is used to perform feature exchange in the channel dimension.

[0061] The spatial interaction layer or channel interaction layer, such as Figure 4 As shown, the calculation formula is as follows:

[0062]

[0063] Where x0 and x1 represent the dual-phase characteristics, x 0 / 1 () represents the spatial interaction layer, where n is the batch size, c is the number of channels in the first or second image to be detected, h is the height of the resolution, w is the width of the resolution, and M is the exchange mask, consisting of 1s and 0s, representing exchange and no exchange, respectively. The channel interaction layer performs feature exchange at the channel layer (c), while the spatial interaction layer performs feature exchange in the spatial dimension (h or w).

[0064] In this embodiment, ground features in the remote sensing image are intertwined and small-scale targets are scattered, leading to the loss of a large amount of detailed information during feature extraction. Therefore, a Local Interaction Enhancement (REX) module is included in the deep encoder to more reliably abstract shallow detailed features into high-level semantic information, while reducing the risk of gradient explosion or gradient vanishing. The structure of this module is as follows: Figure 3As shown. First, a deep encoder is constructed to extract multi-scale local semantic features from the bi-temporal image. The deep encoder contains four layers: the first layer consists of one convolutional layer and one pooling layer; the second, third, and fourth layers consist of 3, 4, and 9 residual blocks, respectively; each residual block contains 3 convolutional layers, which are directly connected. The input high-resolution (resolution greater than or equal to 720) remote sensing bi-temporal image is represented as X. i (i∈{1,2}∈R 3×H×W H and W represent the length and width of the image, respectively, and the number of channels is 3 (RGB). Two depth encoders are used to extract multi-scale local features from the dual-temporal images X1 and X2, respectively. For each X... i Four change features were obtained, which are represented as follows: and Where k = 1, ..., 4, the corresponding feature map size H k W k and dimension C k The corresponding values ​​are [H / 2, W / 2, 64], [H / 4, W / 4, 256], [H / 8, W / 8, 512], and [H / 16, W / 16, 1024]. Simultaneously, local interaction enhancement modules are added after the 2nd, 3rd, and 4th level modules to fuse channel and spatial information between homogeneous features extracted in the previous level, thereby enhancing the similarity between the two branches and achieving data augmentation of the samples. Specifically, spatial interaction layers are added after the 2nd and 4th level modules, and after the 3rd level module. Spatial interaction blocks are used for feature exchange in the spatial dimension, while channel interaction blocks are used for feature exchange in the channel dimension. The feature interaction enhancement blocks use predefined masks to ensure spatial consistency of features. After spatial exchange, the gradient is separated from the exchanged space and backpropagated through other channels. Similarly, the channel interaction blocks perform feature exchange in the channel dimension.

[0065] In step S3, the step of inputting the first local feature map and the second local feature map into the visual self-attention encoder respectively to extract global features and obtain a global feature map includes:

[0066] Step S3.1: Concatenate the first local feature map and the second local feature map to obtain a concatenated feature map;

[0067] Step S3.2: Unfold the cascaded feature map into a two-dimensional image block, and linearly project the two-dimensional image block into a D-dimensional embedding space;

[0068] Step S3.3: Encode the position information of the projected two-dimensional image block to obtain the encoded features;

[0069] Step S3.4: Input the encoded features into the visual self-attention encoder to obtain a global feature map. The visual self-attention encoder includes a multi-head self-attention layer and a multi-layer perceptron layer. The multi-head self-attention layer calculates the attention value of each head and concatenates the attention values ​​of all heads. The concatenated result is used as the output of the multi-head self-attention layer. The attention value of each head is calculated by applying an attention mechanism to the output of each Transformer layer. Each layer of the multi-layer perceptron layer contains a set of non-linear activation functions.

[0070] In this embodiment, although REX can extract local features from remote sensing images, CNN-based methods typically have limitations when modeling long-range dependencies and cannot effectively capture contextual information. The Transformer's self-attention mechanism, however, can effectively focus on the global semantic information of remote sensing images. Therefore, this embodiment designs a Visual Self-Attention Encoder (VIT) after REX to model global contextual information. The Transformer used in this embodiment is the VisionTransformer, with the specific structure as follows... Figure 5 As shown. The first local feature map and the second local feature map output by REX are concatenated to obtain the concatenated feature map feature Y∈R. B×C×H×W The cascaded feature map Y is unfolded into a two-dimensional image patch, represented as follows: in Let B represent the i-th 2D image block, C represent the number of channels in the current 2D image block, and N represent the number of blocks. N = H × W / P 2 Where P represents the block size, and P = 16 is set. A trainable linear projection is used to... Mapping to a D-dimensional embedding space and encoding its positional information to obtain Z0∈R (H×W)×C , represented as:

[0071]

[0072] in, Indicates encoded projection; E pos ∈R N×D This indicates the position code, D = 768.

[0073] The deep feature Z0 obtained after the above processing is input into the VIT, which consists of L Transformer layers, to generate an encoded sequence Z with global context information. The VIT mainly consists of a Multihead Self-Attention (MSA) layer and a Multilayer Perceptron (MLP). The MSA layer, as a key component, is used to capture global context information by simultaneously extracting different feature subspace information carried at different locations in the image patch sequence, thus achieving joint modeling of this information. The design of the MSA layer enables this method to effectively understand and infer the complex dependencies between distant elements in the image, thereby extracting high-order global context features. The MLP is used to enhance the nonlinear expressive power of the model. Each MLP layer contains a series of nonlinear activation functions that can perform depth transformations on the features output by the MSA layer. The introduction of the MLP not only enhances the feature expressive power of the method but also promotes the refined representation of complex remote sensing image change features, helping to improve the overall detection performance. The calculation formulas for the multihead self-attention layer and the multilayer perceptron layer are as follows:

[0074] Z′ l =MSA(LN(Z) l-1 ))+Z l-1 (3)

[0075] Z l =MLP(LN(Z′) l ))+Z′ l (4)

[0076] Among them, Z l-1 Z′ is the encoded feature output by the (l-1)th Transformer layer. l Z represents the features output after computation by a multi-head self-attention layer. l Let L be the encoded feature output by the l-th Transformer layer after multi-layer perceptron computation, where l ∈ {1, ..., L}, L is the total number of layers, and LN(·) is the layer normalization operation, calculated as follows: Where x is the vector representation of the neurons in the l-th layer, g is the scaling parameter, b is the translation parameter, f is the activation parameter, μ is the mean of all neurons in the l-th layer on the training sample, and σ is the variance of all neurons in the l-th layer on the training sample.

[0077] The MSA layer consists of multiple self-attention mechanisms, each capable of capturing different types of contextual information, which helps VIT obtain multi-level, long-distance contextual dependency information. Before calculating the self-attention mechanism, the input Z0 or the output Z of the previous Transformer layer is first processed. lThe mapping is represented by three sets of matrices: a query matrix (Query, Q), a key matrix (Key, K), and a value matrix (Value, V). Next, the correlation weight matrix coefficients between Q and K are calculated. Then, the weight matrix is ​​normalized using a Softmax operation. Finally, the weight coefficients are applied element-wise to V to model the global context information. The calculation process of the self-attention mechanism is as follows:

[0078]

[0079] Where Q, K, and V all have dimensions of N×D, d h Given the dimensions of the Q and K weight matrices, the Transformer structure introduced in this embodiment does not directly use a single self-attention function. Instead, it uses MSA to calculate the attention value for each head separately, and then concatenates the calculation results of the multi-head attention values ​​together as the final output of the MSA layer. The attention value for each head is calculated by applying the attention mechanism to the output of each Transformer layer, as shown in the following formula:

[0080] MSA(Q,K,V)=Concat(head1,head2,…,head n W 0 (6)

[0081] head i =Attention(Z) l-1 W i Q Z l-1 W i K Z l-1 W i V (7)

[0082] Among them, Z l-1 The encoded features are the output of the (l-1)th Transformer layer. For querying the matrix, The key matrix, For value matrices, The parameter matrix of the multi-head self-attention layer, Concat() indicates concatenation, head i Let be the attention value of the i-th self-attention head. Let e ​​be the number of self-attention heads, and d be the attention value of the i-th self-attention head. k =d V =D / e, where D is the dimension of the D-dimensional embedding space.

[0083] In step S4, the process of inputting the global feature map into a cascaded upsampling decoder, performing multiple upsampling and feature fusion operations to obtain a feature map includes:

[0084] Step S4.1: Input the global feature map into multiple upsampling layers for upsampling processing to obtain multiple upsampled features; the upsampling layer includes: one 2x upsampling operator block and two convolutional blocks, the convolutional block includes: a 3×3 convolutional layer, a first batch of normalization layers and a first ReLU activation function, and the upsampling operator block uses bilinear interpolation to perform upsampling;

[0085] Step S4.2: Use skip connections to connect each multi-scale feature of the global feature map with each upsampled feature to obtain the feature map.

[0086] In this embodiment, to balance the complexity and performance of the model, a classic step-by-step upsampling decoder is used to restore the features to their original resolution without compromising depth features, thus achieving decoding processing. Figure 6 As shown. First, the deep features output by VIT are... Reconstructed as Z∈R B×(H / P)×(W / P)×D Then, three upsampling layers are used to restore the features to the original input image size. Furthermore, skip connections are used in CUD to connect each multi-scale feature from the REX output with each upsampled feature, achieving the fusion of shallow and deep features and reducing spatial information loss caused by downsampling. Each upsampling layer consists of one 2x upsampling operator block and two convolutional blocks, where each convolutional block consists of a 3×3 convolutional layer, a BatchNorm (BN) layer, and a ReLU activation function. The upsampling method uses bilinear interpolation for computation, and the cascaded feature dimension of the convolutional layers is consistent with the dimension of the upsampled features. Finally, the multi-scale features extracted in the encoding stage are integrated and decoded to restore spatial details and generate an accurate change detection map.

[0087] In step S5, the classifier includes: two 1×1 convolutional layers, a second ReLU activation function, and a second batch normalization layer.

[0088] In this embodiment, to restore the number of channels in the CUD output feature F to the number of channels in the original image, a classifier is set up. This classifier consists of two 1×1 convolutional layers, one ReLU activation function, and one batch normalization layer connected in series. After the feature map F is processed by this classifier, the final remote sensing image change detection map P is obtained.

[0089] Remote sensing image change detection is a binary classification task, typically calculated using the binary cross-entropy (BCE) loss function. The binary cross-entropy loss function L... bce Represented as:

[0090] Lbce =-t i log(y i )-(1-t i log(1-y) i (8)

[0091] Among them, t i This represents the true label of pixel i. A value of 1 indicates a change category, while a value of 0 indicates an unchanged category. i 1-y represents the probability that the model predicts pixel i belongs to the change category. i This represents the probability that it belongs to an invariant category.

[0092] To describe the specific implementation steps of this embodiment in more detail, a specific example is given below, along with simulation results:

[0093] (1) Parameter settings:

[0094] Parameter settings

[0095] This example demonstrates a semantic fusion-based remote sensing image change detection method implemented using the PyTorch framework. The hardware configuration includes an Intel(R) Xeon(R) Silver 4214 CPU and a Tesla-V100 graphics card. The software configuration uses Python 3.9 and PyTorch 1.12. The hyperparameter settings are as follows: ResNet50 is pre-trained using ImageNet, and the model parameters are updated using the Adam optimizer and the binary cross-entropy loss function. The batch size is set to 12, the learning rate is set to 0.001, and the number of training epochs is 100.

[0096] (2) Example process:

[0097] The specific process for performing change detection on high-resolution remote sensing imagery using this example is as follows:

[0098] Step 1: Divide the input high-resolution remote sensing imagery into a training set and a test set; the training set is used to train a deep encoder with a local interaction enhancement module, a visual self-attention encoder, a cascaded upsampling decoder, and a classifier.

[0099] Step 2: Use a deep encoder with a local interaction enhancement module to extract four multi-scale feature maps F for each bi-temporal image in the test set. i 1 F i 2 F i 3 F i 4 (i∈{1,2}), and will The feature map Y is obtained by cascading the channels;

[0100] Step 3: Input Y into the visual self-attention encoder to extract global contextual relationships and obtain feature map Z;

[0101] Step 4: Input Z into the cascaded upsampling decoder. Through multiple upsampling and feature fusion operations, gradually restore the spatial resolution H×W of the original image to obtain the feature map F.

[0102] Step 5: Upsample the feature map F and input it into the classifier to output the remote sensing image change detection map P;

[0103] (3) Evaluation indicators:

[0104] To evaluate model performance, this example selects F1-score, precision (P), recall (R), and intersection-over-union (IoU) as evaluation metrics. These metrics are defined as follows:

[0105]

[0106] In this context, TP represents the number of correctly detected unchanged pixels; FP represents the number of undetected unchanged pixels; FN represents the number of undetected changed pixels; and TN represents the number of correctly detected changed pixels. For change detection tasks, a higher P-value indicates fewer false positives, and a higher R-value indicates fewer false negatives. The F1-score and IoU metric can also represent the overall performance and generalization ability of the model.

[0107] (4) Comparative experiment:

[0108] This example presents a semantic fusion-based remote sensing image change detection method and experiments it with other methods, including FC-EF (Proceedings of the 2018 25th IEEE International Conference on Image Processing. Athens, Ereece, Caye Daudt et al., 2018), FC-Siam-diff (Proceedings of the 2018 25th IEEE International Conference on Image Processing. Athens, Ereece, Caye Daudt et al., 2018), FC-Siam-conc (Proceedings of the 2018 25th IEEE International Conference on Image Processing. Athens, Ereece, Caye Daudt et al., 2018.), UNet++MSOF (Remote Sensing, Peng et al., 2019), and IFN (Journal of Photogrammetry and Remote Sensing, Zhang et al., 2020). FC-EF employs an image-level fusion method, concatenating two-temporal images into a single input to an FCN. FC-Siam-diff and FC-Siam-conc simultaneously utilize feature-level fusion methods, using a twin FCN to extract multi-level features. FC-Siam-diff fuses two-temporal information using feature difference, while FC-Siam-diff fuses two-temporal information using feature concatenation. UNet++_MSOF, within the UNet++ model, fuses global and fine-grained semantic features through a multi-faceted output fusion strategy, thereby combining change maps at different semantic levels to complete the change detection task. IFN uses a multi-scale feature concatenation method, applying channel attention and spatial attention to the concatenated two-temporal features at each stage of the decoder. Deep supervision (i.e., calculating the supervision loss at each stage of the decoder) is used to better train intermediate layers.

[0109] (5) Experimental analysis and results:

[0110] Table 1 shows the results of the comparative experiments on the LEVIR-CD dataset.

[0111] Table 1 shows the experimental results of change detection on the LEVIR-CD dataset.

[0112]

[0113] As shown in Table 1, the method presented in this example achieves an accuracy (P) of 92.72% and a recall (R) of 87.01% on the LEVIR-CD dataset, with an F1 score of 89.22% and an IoU of 81.02%. Compared to the classification results of FC-EF, FC-Siam-diff, FC-Siam-conc, UNet++_MSOF, and IFN methods, the method shows improvements in almost all aspects. For example, the accuracy (P) is improved by approximately 1.75% to 7.24%, indicating that introducing the Transformer to obtain rich global contextual information effectively enhances the model's ability to identify change detection regions. In summary, the method proposed in this example can adaptively mine remote sensing feature map contextual information from different perspectives, enabling accurate label prediction and demonstrating good detection performance for objects at multiple scales.

[0114] To better describe the experimental results, a visual analysis diagram of the results is presented using the experimental data in Table 1. Figure 7 , Figure 8 ).like Figure 7 As shown, the FC-EF, FC-Siam-diff, FC-Siam-conc, UNet++_MSOF, and IFN methods will result in some missed building detections, as illustrated in the first and fourth rows of the results. It can be seen that the method in this example can capture multi-level adaptive scaling feature maps, achieving a better building detection map. This is because the REX module utilizes the spatial and channel correlation relationships between features to obtain more accurate features, making building change detection more accurate. Simultaneously, due to the introduction of the VIT module, appropriate contextual information is extracted for objects, adaptively enhancing feature representation capabilities, which not only reduces incomplete and irregular semantic objects but also better preserves geometric details and complex contours. Figure 7 In the first, second, and fourth rows, spatial location information is preserved. This indicates that the method significantly improves the detection of changes in remote sensing images.

[0115] This embodiment proposes a remote sensing image change detection method based on semantic fusion, which combines the core advantages of Transformer and convolutional neural networks. First, a deep encoder is designed to encode local pixel blocks in dual-temporal remote sensing images. Simultaneously, a Transformer module is introduced to further extract and integrate global contextual information of the image. In the decoding stage, upsampling techniques are used to gradually restore the encoded high-level semantic features to the spatial resolution of the original image. The decoder employs a skip connection mechanism to concatenate multi-scale local detail features with global contextual features, achieving effective fusion of local and global semantic features. This fusion approach can enhance the sensitivity and accuracy of detecting changes in various land features while maintaining accurate location of changed areas, thereby improving the method's adaptability and robustness in various scenarios.

[0116] Example 2:

[0117] This embodiment proposes a remote sensing image change detection system based on semantic fusion, such as... Figure 9 As shown, it includes: an image acquisition module, a local feature extraction module, a global feature extraction module, an upsampling decoding module, and an image classification module, which are connected sequentially.

[0118] The image acquisition module is used to acquire multiple batches of first images to be detected and second images to be detected, wherein the first images to be detected and the second images to be detected are dual-temporal images of the same region;

[0119] The local feature extraction module is used to input the first image to be detected and the second image to be detected into the depth encoder with the local interaction enhancement module, respectively, to extract local features and obtain the first local feature map and the second local feature map.

[0120] The global feature extraction module is used to input the first local feature map and the second local feature map into the visual self-attention encoder respectively to extract global features and obtain a global feature map.

[0121] The upsampling decoding module is used to input the global feature map into the cascaded upsampling decoder, perform multiple upsampling and feature fusion operations to obtain the feature map;

[0122] The image classification module is used to input the feature map into the classifier for classification, and obtain the remote sensing image change detection map.

[0123] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0124] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A method for detecting changes in remote sensing images based on semantic fusion, characterized in that, include: Multiple batches of first and second images to be detected are acquired, wherein the first and second images to be detected are dual-temporal images of the same region. The first image to be detected and the second image to be detected are respectively input into a depth encoder with a local interaction enhancement module to extract local features and obtain a first local feature map and a second local feature map respectively. The first local feature map and the second local feature map are respectively input into the visual self-attention encoder to extract global features and obtain a global feature map; The global feature map is input into a cascaded upsampling decoder, and after multiple upsampling and feature fusion operations, a feature map is obtained. The feature map is input into a classifier for classification to obtain a remote sensing image change detection map. The deep encoder with local interaction enhancement modules includes: a first layer, a second layer, a third layer, and a fourth layer. Local interaction enhancement modules are set between the second and third layers, between the third and fourth layers, and after the fourth layer. The first layer includes: a convolutional layer and a pooling layer; the second layer includes: a first residual module and three second residual modules; the third layer includes: a first residual module and four second residual modules; the fourth layer includes: a first residual module and nine second residual modules. The local interaction enhancement modules include: a spatial interaction layer and a channel interaction layer. A spatial interaction layer is set between the second and third layers, a channel interaction layer is set between the third and fourth layers, and a spatial interaction layer is set after the fourth layer. The spatial interaction layer is used for feature exchange in the spatial dimension, and the channel interaction layer is used for feature exchange in the channel dimension. The spatial interaction layer or channel interaction layer is calculated as follows: ； in, , Indicates dual-phase characteristics, () represents the spatial interaction layer, n represents the number of batches, c represents the number of channels of the first or second image to be detected, h represents the height of the resolution, w represents the width of the resolution, and M represents the exchange mask, which consists of 1 and 0, where 1 indicates exchange and 0 indicates no exchange.

2. The remote sensing image change detection method based on semantic fusion according to claim 1, characterized in that, The step of inputting the first local feature map and the second local feature map into a visual self-attention encoder to extract global features and obtain a global feature map includes: The first local feature map and the second local feature map are concatenated to obtain a concatenated feature map. The cascaded feature map is unfolded into a two-dimensional image block, and the two-dimensional image block is linearly projected into a D-dimensional embedding space; The positional information of the projected two-dimensional image block is encoded to obtain the encoded features; The encoded features are input into a visual self-attention encoder to obtain a global feature map. The visual self-attention encoder includes a multi-head self-attention layer and a multi-layer perceptron layer. The multi-head self-attention layer calculates the attention value of each head and concatenates the attention values ​​of all heads. The concatenated result is used as the output of the multi-head self-attention layer. The attention value of each head is calculated by applying an attention mechanism to the output of each Transformer layer. Each layer of the multi-layer perceptron layer contains a set of non-linear activation functions.

3. The remote sensing image change detection method based on semantic fusion according to claim 2, characterized in that, The calculation formulas for the multi-head self-attention layer and the multi-layer perceptron layer are as follows: ； ； in, For the first l-1 The encoded features output by each Transformer layer The features are the output after computation by the multi-head self-attention layer. For the first layer of computation after multiple perceptron layers l The encoded features output by each Transformer layer L represents the total number of layers in the visual self-attention encoder. For layer normalization operations, the calculation formula is: Where x is the th l The vector representation of a layer of neurons, where g is the scaling parameter, b is the translation parameter, and f is the activation parameter. For the training samples l The mean of all neurons in the layer, For the training samples l The variance of all neurons in the layer.

4. The remote sensing image change detection method based on semantic fusion according to claim 2, characterized in that, The multi-head self-attention layer calculates the attention value for each head separately, and concatenates the attention values ​​of all heads. The concatenated result is used as the output of the multi-head self-attention layer. The attention value of each head is calculated by applying an attention mechanism to the output of each Transformer layer, as shown in the following formula: ； ； in, For the first l-1 The encoded features output by each Transformer layer For querying the matrix, The key matrix, For value matrices, The parameter matrix of the multi-head self-attention layer, where Concat() represents the cascading operation. Let be the attention value of the i-th self-attention head, and e be the number of self-attention heads. = =D / e, where D is the dimension of the D-dimensional embedding space.

5. The remote sensing image change detection method based on semantic fusion according to claim 1, characterized in that, The step of inputting the global feature map into a cascaded upsampling decoder, performing multiple upsampling and feature fusion operations to obtain a feature map includes: The global feature map is input into multiple upsampling layers for upsampling processing to obtain multiple upsampling features. Each upsampling layer includes: one 2x upsampling operator block and two convolutional blocks. Each convolutional block includes: a 3×3 convolutional layer, a first batch of normalization layers, and a first ReLU activation function. The upsampling operator block uses bilinear interpolation to perform upsampling. By using skip connections, each multi-scale feature of the global feature map is connected to each upsampled feature to obtain the feature map.

6. The remote sensing image change detection method based on semantic fusion according to claim 1, characterized in that, The classifier includes: two 1×1 convolutional layers, a second ReLU activation function, and a second batch normalization layer.

7. A remote sensing image change detection system based on semantic fusion, characterized in that, include: The image acquisition module is used to acquire multiple batches of first images to be detected and second images to be detected, wherein the first images to be detected and the second images to be detected are dual-temporal images of the same region. The local feature extraction module is used to input the first image to be detected and the second image to be detected into the depth encoder with the local interaction enhancement module, respectively, to extract local features and obtain the first local feature map and the second local feature map. The global feature extraction module is used to input the first local feature map and the second local feature map into the visual self-attention encoder respectively to extract global features and obtain a global feature map. The upsampling decoding module is used to input the global feature map into the cascaded upsampling decoder, and obtain the feature map after multiple upsampling and feature fusion operations; The image classification module is used to input the feature map into the classifier for classification to obtain a remote sensing image change detection map. The deep encoder with local interaction enhancement modules includes a first layer, a second layer, a third layer, and a fourth layer. Local interaction enhancement modules are placed between the second and third layers, between the third and fourth layers, and after the fourth layer. The first layer includes convolutional layers and pooling layers; the second layer includes a first residual module and three second residual modules; the third layer includes a first residual module and four second residual modules; and the fourth layer includes a first residual module and nine second residual modules. The local interaction enhancement module includes a spatial interaction layer and a channel interaction layer. A spatial interaction layer is set between the second and third levels, a channel interaction layer is set between the third and fourth levels, and a spatial interaction layer is set after the fourth level. The spatial interaction layer is used to perform feature exchange in the spatial dimension, and the channel interaction layer is used to perform feature exchange in the channel dimension. The spatial interaction layer or channel interaction layer is calculated as follows: ； in, , Indicates dual-phase characteristics, () represents the spatial interaction layer, n represents the number of batches, c represents the number of channels of the first or second image to be detected, h represents the height of the resolution, w represents the width of the resolution, and M represents the exchange mask, which consists of 1 and 0, where 1 indicates exchange and 0 indicates no exchange.

Images