Remote sensing image change detection method and system based on semantic fusion
By combining the Transformer model with convolutional neural network, multi-scale local details and global context features of remote sensing images are extracted, and feature fusion is performed in the decoder, which solves the insufficient accuracy and generalization ability of remote sensing images change detection in the existing technology. Problem, high sensitivity and accurate change detection are achieved.
Patent Information
- Application Number
- CN202411105153.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-13
AI Technical Summary
The existing remote sensing image change detection methods have pseudo-change, scale change and category imbalance in high-resolution images. The traditional method relies on manual feature extraction, making it difficult to achieve high-change detection accuracy and generalization capabilities. CNN context within the spatial range Information modeling is limited, and the Transformer model ignores local information to increase computational complexity.
Using a semantic fusion-based method, combining the Transformer model with a convolutional neural network, multi-scale local detail features and global context features are extracted through local interaction enhancement modules and visual self-attention encoder, and cascade upsampling and Feature fusion realizes effective fusion of local and global semantic features.
It improves the sensitivity and accuracy of remote sensing image change detection, enhances the adaptability and robustness of the method in various scenarios, and can accurately identify the changing areas and complete the boundaries.
Smart Images

Figure CN119068351A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of remote sensing image data processing and application, and in particular relates to a remote sensing image change detection method and system based on semantic fusion. Background Art
[0002] Change detection (CD) in remote sensing images refers to the process of detecting surface changes based on georeferenced multi-temporal remote sensing images. As one of the important technologies for quickly and effectively acquiring changes in remote sensing observations, CD of remote sensing images has attracted extensive attention in many remote sensing-related tasks, such as land use surveys, urban planning, environmental monitoring, and disaster mapping. Since high-resolution remote sensing images can provide rich spatial information and fine surface details, detecting their changes is of great significance for fine-grained applications such as precision agriculture and urban renewal. However, CD in high-resolution remote sensing images faces many challenges, such as pseudo-changes, scale changes, and class imbalance. Traditional methods mostly rely on artificial feature extraction, such as principal component analysis (PCA), Gabor filtering, multivariate alteration detection (MAD), and change vector analysis (CVA). These methods can achieve CD tasks to a certain extent, but they have the following two disadvantages: on the one hand, the image features extracted by traditional methods are easily affected by seasonal changes, lighting conditions and satellite sensors, making it difficult to achieve high change detection accuracy; on the other hand, the manually extracted features are heavily dependent on domain prior knowledge, which limits the generalization ability of CD methods.
[0003] The rise of deep learning technology has led to the proposal of many new CD methods, among which Convolutional Neural Network (CNN) has strong discriminative ability and has been successfully applied to CD tasks. CNN-based methods convert dual-phase images into high-level spatial depth features and extract highly abstract contextual semantics of the change area, which minimizes error propagation and reduces the human error caused by preprocessing. However, since CNN can only realize the extraction of global information from local information reasoning by gradually expanding the receptive field, the current CNN-based CD methods cannot fully realize the modeling of contextual information in the spatial range. The latest research on remote sensing image CD shows that the Transformer model (a deep neural network model based on the self-attention mechanism) breaks through the limitation of the local receptive field of CNN and can provide long-distance dependencies for semantic changes. The Transformer model has achieved excellent results in applications such as image classification, object detection, and semantic segmentation in the field of computer vision, but using only the Transformer as an encoder will ignore the importance of local information and increase the complexity of the calculation. Summary of the invention
[0004] In view of the shortcomings of the existing technology, this application proposes a remote sensing image change detection method and system based on semantic fusion, which integrates the Transformer model and the convolutional neural network, and uses a skip connection mechanism in the decoder to cascade multi-scale local detail features with global context features to achieve effective fusion of local and global semantic features. The fusion method proposed in this application can enhance the sensitivity and accuracy of change detection of various types of objects while maintaining accurate positioning of the change area, thereby improving the adaptability and robustness of the method in various scenarios.
[0005] In the first aspect, the present application proposes a remote sensing image change detection method based on semantic fusion, comprising:
[0006] Acquire multiple batches of first images to be detected and second images to be detected, wherein the first images to be detected and the second images to be detected are dual-phase images of the same area;
[0007] Inputting the first image to be detected and the second image to be detected into a deep encoder with a local interaction enhancement module respectively, extracting local features, and obtaining a first local feature map and a second local feature map respectively;
[0008] Inputting the first local feature map and the second local feature map into a visual self-attention encoder respectively, extracting global features, and obtaining a global feature map;
[0009] The global feature map is input into a cascade upsampling decoder, and a feature map is obtained after multiple upsampling and feature fusion operations;
[0010] The feature map is input into a classifier for classification to obtain a remote sensing image change detection map.
[0011] The deep encoder with a local interaction enhancement module includes: a first level, a second level, a third level and a fourth level, and the local interaction enhancement module is arranged between the second level and the third level, between the third level and the fourth level and after the fourth level, the first level includes: a convolution layer and a pooling layer; the second level includes: a first residual module and 3 second residual modules; the third level includes: a first residual module and 4 second residual modules; the fourth level includes: a first residual module and 9 second residual modules.
[0012] The local interaction enhancement module includes: a spatial interaction layer and a channel interaction layer. The spatial interaction layer is set between the second level and the third level, the channel interaction layer is set between the third level and the fourth level, and the spatial interaction layer is set after the fourth level. The spatial interaction layer is used to exchange features in the spatial dimension, and the channel interaction layer is used to exchange features in the channel dimension.
[0013] The spatial interaction layer or channel interaction layer is calculated as follows:
[0014]
[0015] Among them, x0 and x1 represent the dual-phase characteristics, x 0 / 1 () is the spatial interaction layer, n is the number of batches, c is the number of channels of the first image to be detected or the second image to be detected, h is the height of the resolution, w is the width of the resolution, and M is the exchange mask, which consists of 1 and 0, 1 means exchange, and 0 means no exchange.
[0016] The step of inputting the first local feature map and the second local feature map into a visual self-attention encoder respectively, extracting global features, and obtaining a global feature map comprises:
[0017] Cascading the first local feature map and the second local feature map to obtain a cascade feature map;
[0018] Expanding the cascade feature map into a two-dimensional image block, and linearly projecting the two-dimensional image block into a D-dimensional embedding space;
[0019] Encoding the position information of the projected two-dimensional image block to obtain a coding feature;
[0020] The encoded features are input into a visual self-attention encoder to obtain a global feature map. The visual self-attention encoder includes: a multi-head self-attention layer and a multi-layer perceptron layer. The multi-head self-attention layer calculates the attention value of each head respectively, and concatenates the attention values of all heads. The concatenated result is used as the output of the multi-head self-attention layer. The attention value of each head is calculated by performing an attention mechanism on the output of each Transformer layer. Each layer of the multi-layer perceptron layer includes a set of nonlinear activation functions.
[0021] The multi-head self-attention layer and the multi-layer perceptron layer are calculated as follows:
[0022] Z′ l =MSA(LN(Z l-1 ))+Z l-1
[0023] Z l =MLP(LN(Z′ l ))+Z′ l
[0024] Among them, Z l-1 is the encoded feature output by the l-1th Transformer layer, Z′ l is the feature output after calculation by the multi-head self-attention layer, Z l is the encoded feature output by the lth Transformer layer after calculation by the multi-layer perceptron layer, l∈{1,…,L}, L is the total number of layers of the visual self-attention encoder, and LN(·) is the layer normalization operation, which is calculated as: Among them, x is the vector representation of the neurons in the lth layer, g is the scaling parameter, b is the translation parameter, f is the activation parameter, μ is the mean of all neurons in the lth layer on the training sample, and σ is the variance of all neurons in the lth layer on the training sample.
[0025] The multi-head self-attention layer calculates the attention value of each head respectively, and concatenates the attention values of all heads, and uses the concatenated result as the output of the multi-head self-attention layer. The attention value of each head is calculated by the attention mechanism of the output of each Transformer layer, and the calculation formula is as follows:
[0026] MSA(Q,K,V)=Concat(head1,head2,…,head n )W 0
[0027] head i =Attention(Z l-1 W i Q , Zl-1 W i K , Z l-1 W i V )
[0028] Among them, Z l-1 is the encoding feature output by the l-1th Transformer layer, is the query matrix, is the key matrix, is the value matrix, The parameter matrix of the multi-head self-attention layer, Concat() represents the cascade operation, head i is the attention value of the i-th self-attention head, e is the number of self-attention heads, and d k =d v =D / e, where D is the dimension of the D-dimensional embedding space.
[0029] The step of inputting the global feature map into a cascade upsampling decoder, performing multiple upsampling and feature fusion operations to obtain a feature map includes:
[0030] Inputting the global feature map into multiple upsampling layers for upsampling processing to obtain multiple upsampling features; the upsampling layer includes: 1 2x upsampling operator block and 2 convolution blocks, the convolution block includes: 3×3 convolution layer, the first batch of normalization layers and the first ReLU activation function, and the upsampling operator block uses bilinear interpolation calculation for upsampling;
[0031] A skip connection is used to connect each multi-scale feature of the global feature map with each upsampled feature to obtain a feature map.
[0032] The classifier includes: 2 1×1 convolutional layers, a second ReLU activation function and a second batch normalization layer.
[0033] In the second aspect, the present application proposes a remote sensing image change detection system based on semantic fusion, comprising:
[0034] An image acquisition module, used for acquiring a first image to be detected and a second image to be detected in batches, wherein the first image to be detected and the second image to be detected are dual-phase images of the same area;
[0035] A local feature extraction module, used to input the first image to be detected and the second image to be detected into a depth encoder with a local interaction enhancement module, extract local features, and obtain a first local feature map and a second local feature map respectively;
[0036] A global feature extraction module, used for inputting the first local feature map and the second local feature map into a visual self-attention encoder respectively, extracting global features, and obtaining a global feature map;
[0037] An upsampling decoding module is used to input the global feature map into a cascade upsampling decoder, perform multiple upsampling and feature fusion operations, and obtain a feature map;
[0038] The image classification module is used to input the feature map into a classifier for classification to obtain a remote sensing image change detection map.
[0039] Beneficial effects:
[0040] This application proposes a remote sensing image change detection method and system based on semantic fusion. First, a deep encoder with a local interaction enhancement module is used to encode the pixel blocks of the dual-phase remote sensing image, and then a visual self-attention encoder is introduced to deeply mine the change features extracted by the deep encoder to fully explore and integrate the global context association information of the image. Finally, in the decoding stage, the upsampling technology is used to gradually restore the encoded high-level semantic features to the original spatial resolution size of the image, realizing the effective fusion of local and global semantic features, so that the change area can be accurately identified and the boundary of the change area is more complete. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 A flow chart of a remote sensing image change detection method based on semantic fusion according to an embodiment of the present application;
[0042] Figure 2 A flowchart of a remote sensing image change detection method based on semantic fusion according to an embodiment of the present application;
[0043] Figure 3 A schematic diagram of the structure of a deep encoder with a local interaction enhancement module according to an embodiment of the present application;
[0044] Figure 4 Schematic diagram of channel interaction in an embodiment of the present application;
[0045] Figure 5 Schematic diagram of the structure of the visual self-attention encoder in the embodiment of the present application;
[0046] Figure 6 Schematic diagram of the structure of the cascaded upsampling decoder according to the embodiment of the present application
[0047] Figure 7 Schematic diagram of the visual analysis results of the embodiment of the present application;
[0048] Figure 8 Schematic diagram of the visual analysis result flow of the embodiment of the present application;
[0049] Fig. 9 A principle block diagram of a remote sensing image change detection system based on semantic fusion in an embodiment of the present application. DETAILED DESCRIPTION
[0050] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.
[0051] The ability to extract both local and global features in CD methods is very important and has received extensive attention in recent studies. CNN has become the mainstream framework for extracting local features of remote sensing images in many advanced CD methods. The Transformer model provides an effective method for extracting global features from remote sensing images. This application proposes a remote sensing image change detection method and system based on semantic fusion, the core components of which include a local interaction enhancement module (REX, Real-Time Executive), a visual self-attention encoder (VIT, Vision Transformer) and a cascaded upsampling decoder (CUD, Cascaded Upsampling Decoder). In the encoding stage, the dual-phase image is subjected to a weight-sharing local interaction enhancement module to extract multi-scale local semantic features and realize feature information interaction in space and channels. Then, the REX output features are channel-joined with the dual-phase semantic features and input into VIT to extract global semantic features, thereby realizing the fusion of local and global features of the dual-phase image. In the decoding stage, a decoder CUD containing multiple upsampling blocks is designed to decode hidden features and gradually restore the features to the original image scale.
[0052] Embodiment 1:
[0053] This embodiment proposes a remote sensing image change detection method based on semantic fusion, such as Figure 1 , Figure 2 As shown, including:
[0054] Step S1: Acquire a first image to be detected and a second image to be detected, wherein the first image to be detected and the second image to be detected are dual-phase images of the same area;
[0055] Step S2: inputting the first image to be detected and the second image to be detected into a depth encoder with a local interaction enhancement module respectively, extracting local features, and obtaining a first local feature map and a second local feature map respectively;
[0056] Step S3: inputting the first local feature map and the second local feature map into a visual self-attention encoder respectively, extracting global features, and obtaining a global feature map;
[0057] Step S4: inputting the global feature map into a cascade upsampling decoder, and obtaining a feature map after multiple upsampling and feature fusion operations;
[0058] Step S5: input the feature map into a classifier for classification to obtain a remote sensing image change detection map.
[0059] In step 2, the deep encoder with a local interaction enhancement module includes: a first level, a second level, a third level and a fourth level, and a local interaction enhancement module is set between the second level and the third level, between the third level and the fourth level, and after the fourth level, the first level includes: a convolution layer and a pooling layer; the second level includes: a first residual module and 3 second residual modules; the third level includes: a first residual module and 4 second residual modules; the fourth level includes: a first residual module and 9 second residual modules.
[0060] The local interaction enhancement module includes: a spatial interaction layer and a channel interaction layer. The spatial interaction layer is set between the second level and the third level, the channel interaction layer is set between the third level and the fourth level, and the spatial interaction layer is set after the fourth level. The spatial interaction layer is used to exchange features in the spatial dimension, and the channel interaction layer is used to exchange features in the channel dimension.
[0061] The spatial interaction layer or channel interaction layer, such as Figure 4 As shown, the calculation formula is as follows:
[0062]
[0063] Among them, x0 and x1 represent the dual-phase characteristics, x 0 / 1 () is the spatial interaction layer, n is the number of batches, c is the number of channels of the first image to be detected or the second image to be detected, h is the height of the resolution, w is the width of the resolution, and M is the exchange mask, which consists of 1 and 0, indicating exchange and no exchange, respectively. The channel interaction layer performs feature exchange in the channel layer (c), and the spatial interaction layer performs feature exchange in the spatial dimension (that is, h or w).
[0064] In this embodiment, the ground objects in the remote sensing image are intertwined and the small-scale objects are scattered, resulting in a large amount of detail information being lost during the feature extraction process. Therefore, the local interaction enhancement module (REX) is included in the deep encoder to more reliably abstract the shallow detail features into high-level semantic information, while reducing the risk of gradient explosion or gradient disappearance. The module structure is as follows Figure 3As shown in Figure 1. First, a deep encoder is constructed to extract multi-scale local semantic features of dual-temporal images. The deep encoder contains 4 layers of modules; the first layer consists of 1 convolution layer and 1 pooling layer, and the second, third and fourth layers are composed of 3, 4 and 9 residual blocks respectively; each residual block contains 3 convolution layers and is connected in a direct connection. The input high-resolution (resolution greater than or equal to 720 images) remote sensing dual-temporal image is represented by X i (i∈{1,2}∈R 3×H×W ), H and W represent the length and width of the image respectively, and the number of channels is 3 (RGB). Two deep encoders are used to extract multi-scale local features of the dual-phase images X1 and X2 respectively. i Four change features are obtained, which are expressed as and Where k = 1, ..., 4, the corresponding feature map size H k , W k and dimension C k They are [H / 2, W / 2, 64], [H / 4, W / 4, 256], [H / 8, W / 8, 512] and [H / 16, W / 16, 1024] respectively. At the same time, local interaction enhancement modules are added after the 2nd, 3rd and 4th level modules respectively to fuse the channel information and spatial information between homogeneous features in the feature extraction process of the previous level to enhance the similarity of the features of the two branches and achieve data enhancement of the samples. Specifically, a spatial interaction layer is added after the 2nd and 4th level modules, and a spatial interaction layer is added after the 3rd level module. The spatial interaction block is used to exchange features in the spatial dimension, and the channel interaction block is used to exchange features in the channel dimension. The feature interaction enhancement block uses a predefined mask to ensure the spatial consistency of the features. After the spatial exchange, the gradient is separated from the exchanged space and back-propagated through other channels. Similarly, the channel interaction block performs feature exchange in the channel dimension.
[0065] In step S3, the first local feature map and the second local feature map are respectively input into a visual self-attention encoder to extract global features to obtain a global feature map, including:
[0066] Step S3.1: cascading the first local feature map and the second local feature map to obtain a cascade feature map;
[0067] Step S3.2: Expand the cascade feature map into a two-dimensional image block, and linearly project the two-dimensional image block into a D-dimensional embedding space;
[0068] Step S3.3: encoding the position information of the projected two-dimensional image block to obtain a coding feature;
[0069] Step S3.4: Input the encoded features into the visual self-attention encoder to obtain a global feature map. The visual self-attention encoder includes: a multi-head self-attention layer and a multi-layer perceptron layer. The multi-head self-attention layer calculates the attention value of each head respectively, and concatenates the attention values of all heads. The concatenated result is used as the output of the multi-head self-attention layer. The attention value of each head is calculated by performing an attention mechanism on the output of each Transformer layer; each layer of the multi-layer perceptron layer contains a set of nonlinear activation functions.
[0070] In this embodiment, although REX can extract local features of remote sensing images, CNN-based methods are usually limited in modeling long-distance dependencies and cannot effectively capture contextual information. The self-attention mechanism of Transformer can focus on the global semantic information of remote sensing images. Therefore, this embodiment designs a visual self-attention encoder (VIT) after REX to model global contextual information. The Transformer used in this embodiment is VisionTransformer, and its specific structure is as follows: Figure 5 As shown. The first local feature map output by REX is cascaded with the second local feature map to obtain the cascade feature map feature Y∈R B×C×H×W , expand the cascade feature map Y into a two-dimensional image block, expressed as in represents the i-th two-dimensional image block, B represents the batch size, C represents the number of channels of the current two-dimensional image block, and N represents the number of blocks, N = H × W / P 2 , where P represents the size of the block, set P = 16. Using trainable linear projection Map it into a D-dimensional embedding space and encode its position information to obtain Z0∈R (H×W)×C , expressed as:
[0071]
[0072] in, represents the coded projection; E pos ∈R N×D Indicates position coding, D=768.
[0073] The deep feature Z0 obtained after the above processing is input to VIT composed of L Transformer layers to generate a coding sequence Z with global context information. VIT is mainly composed of a multi-head self-attention (MSA) layer and a multi-layer perceptron (MLP). The MSA layer is a key component used to capture global context information. By simultaneously extracting different feature subspace information carried by different positions in the image block sequence, the joint modeling of these information is realized. The design of the MSA layer enables this method to effectively understand and reason about the complex dependencies between long-distance elements in the image, thereby extracting high-order global context features. MLP is used to enhance the nonlinear expression ability of the model. Each layer of MLP contains a series of nonlinear activation functions, which can perform deep transformations on the features output by the MSA layer. The introduction of MLP can not only enhance the feature expression ability of the method, but also promote the method to refine the characteristics of complex remote sensing image changes, which helps to improve the overall detection performance. The multi-head self-attention layer and the multi-layer perceptron layer are calculated as follows:
[0074] Z′ l =MSA(LN(Z l-1 ))+Z l-1 (3)
[0075] Z l =MLP(LN(Z′ l ))+Z′ l (4)
[0076] Among them, Z l-1 is the encoded feature output by the l-1th Transformer layer, Z′ l is the feature output after calculation by the multi-head self-attention layer, Z l is the encoding feature output by the lth Transformer layer after calculation by the multi-layer perceptron layer, l∈{1,…,L}, L is the total number of layers, LN(·) is the layer normalization operation, and the calculation formula is: Among them, x is the vector representation of the neurons in the lth layer, g is the scaling parameter, b is the translation parameter, f is the activation parameter, μ is the mean of all neurons in the lth layer on the training sample, and σ is the variance of all neurons in the lth layer on the training sample.
[0077] The MSA layer consists of multiple self-attention mechanisms, each of which can capture different types of contextual information, helping VIT obtain multi-level long-distance contextual dependency information. Before calculating the self-attention mechanism, first convert the input Z0 or the output result Z of the previous Transformer layer into lIt is mapped into three sets of matrices, namely the query matrix (Query, Q), the key matrix (Key, K) and the value matrix (Value, V). Secondly, the correlation weight matrix coefficients between Q and K are calculated. Thirdly, the weight matrix is normalized by the Softmax operation. Finally, the weight coefficients are applied element by element to V to achieve modeling of global context information. The calculation process of the self-attention mechanism is as follows:
[0078]
[0079] Among them, the size of Q, K and V is N×D, d h is the dimension of the Q and K weight matrices. The Transformer structure introduced in this embodiment does not directly use a single self-attention function, but uses MSA to calculate the attention value of each head respectively, and then splices the calculation results of the multi-head attention values together as the final output of the MSA layer. The attention value of each head is calculated by the attention mechanism of the output of each Transformer layer, and the calculation formula is as follows:
[0080] MSA(Q,K,V)=Concat(head1,head2,…,head n )W 0 (6)
[0081] head i =Attention(Z l-1 W i Q , Z l-1 W i K , Z l-1 W i V ) (7)
[0082] Among them, Z l-1 is the encoding feature output by the l-1th Transformer layer, is the query matrix, is the key matrix, is the value matrix, The parameter matrix of the multi-head self-attention layer, Concat() represents cascade, head i is the attention value of the ith self-attention head. e is the number of self-attention heads, and d k =d V =D / e, where D is the dimension of the D-dimensional embedding space.
[0083] In step S4, the global feature map is input into a cascade upsampling decoder, and multiple upsampling and feature fusion operations are performed to obtain a feature map, including:
[0084] Step S4.1: inputting the global feature map into multiple upsampling layers for upsampling processing to obtain multiple upsampling features; the upsampling layer includes: 1 2x upsampling operator block and 2 convolution blocks, the convolution block includes: 3×3 convolution layer, the first batch of normalization layers and the first ReLU activation function, and the upsampling operator block uses bilinear interpolation calculation for upsampling;
[0085] Step S4.2: Use skip connection to connect each multi-scale feature of the global feature map with each upsampled feature to obtain a feature map.
[0086] In this embodiment, in order to balance the complexity and performance of the model, a classic step-by-step upsampling decoder is used to restore the features to the original resolution without destroying the deep features as much as possible to implement decoding processing, such as Figure 6 As shown. First, the deep features output by VIT Reconstructed as Z∈R B×(H / P)×(W / P)×D . Then, three upsampling layers are used to restore the features to the size of the original input image. In addition, skip connections are used in CUD to connect each multi-scale feature output by REX with each upsampled feature to achieve the fusion of shallow and deep features and reduce the loss of spatial information caused by downsampling. Each upsampling layer consists of a 2x upsampling operator block and 2 convolution blocks, where each convolution block consists of a 3×3 convolution layer, a BatchNorm (BN) layer, and a ReLU activation function. The upsampling method uses bilinear interpolation for calculation, and the cascade feature dimension of the convolution layer is consistent with the dimension of the upsampled features. Finally, the multi-scale features extracted in the encoding stage are integrated and decoded to restore spatial details to generate accurate change detection maps.
[0087] In step S5, the classifier includes: 2 1×1 convolutional layers, a second ReLU activation function and a second batch normalization layer.
[0088] In this embodiment, in order to restore the number of channels of the CUD output feature F to the number of channels of the original image, the present invention sets a classifier. The classifier is composed of two 1×1 convolutional layers, one ReLU activation function and one BatchNormalization layer connected in series. After the feature map F is processed by this classifier, the final remote sensing image change detection map P is obtained.
[0089] Remote sensing image change detection is a binary classification task, which is usually calculated using the binary cross-entropy loss function (BCE). bce It is expressed as:
[0090] Lbce =-t i log(y i )-(1-t i )log(1-y i ) (8)
[0091] Among them, t i Represents the true label of pixel i. A value of 1 corresponds to a changed category, and a value of 0 indicates an unchanged category. i Represents the probability that the model predicts that pixel i belongs to the change category, 1-y i It represents the probability of predicting that it belongs to the unchanged category.
[0092] In order to describe the specific implementation steps of this embodiment in more detail, a specific example is given and simulation experiment results are given:
[0093] (1) Parameter setting:
[0094] Parameter settings
[0095] In this example, a remote sensing image change detection method based on semantic fusion is implemented based on the Pytorch framework. The hardware configuration is Intel(R) Xeon(R) Silver 4214CPU, Tesla-V100 graphics card, and the software configuration is Python3.9+Pytorch1.12 version. The hyperparameter settings are as follows: ResNet50 uses ImageNet as the pre-training calculation, and the model parameters are updated by the Adam optimizer and the binary cross entropy loss function. The batch size is set to 12, the learning rate is set to 0.001, and the number of training rounds is 100.
[0096] (2) Example process:
[0097] The specific process of using this example to perform change detection on high-resolution remote sensing images is as follows:
[0098] Step 1: Divide the input high-resolution remote sensing image into a training set and a test set; the training set is used to train the deep encoder with local interaction enhancement module, the visual self-attention encoder, the cascade upsampling decoder and the classifier;
[0099] Step 2: Use a deep encoder with a local interaction enhancement module to extract four multi-scale feature maps F for each dual-phase image in the test set i 1 , F i 2 , F i 3 , F i 4 (i∈{1,2}), and Perform channel cascading to obtain feature map Y;
[0100] Step 3: Input Y into the visual self-attention encoder to extract the global context relationship and obtain the feature map Z;
[0101] Step 4: Input Z into the cascade upsampling decoder, and gradually restore the spatial resolution H×W of the original image through multiple upsampling and feature fusion operations to obtain the feature map F;
[0102] Step 5: Upsample the feature map F and input it into the classifier to output the remote sensing image change detection map P;
[0103] (3) Evaluation indicators:
[0104] To test the performance of the model, this example selects F1-score, precision (P), recall (R), and intersection over union (IoU) as evaluation indicators. The above indicators are defined as follows:
[0105]
[0106] Among them, TP represents the number of unchanged pixels correctly detected; FP represents the number of unchanged pixels not detected; FN represents the number of changed pixels not detected; TN represents the number of changed pixels correctly detected. For the change detection task, the higher the P value, the fewer the number of false detections; the higher the R value, the fewer the number of missed detections; the F1-score value and IoU index can also reflect the overall performance and generalization ability of the model.
[0107] (4) Comparative experiment:
[0108] This example proposes a remote sensing image change detection method based on semantic fusion and experiments with other methods, including FC-EF (Proceedings of the 2018 25th IEEE International Conference on Image Processing. Athens, Ereece, Caye Daudt et al., 2018), FC-Siam-diff (Proceedings of the 2018 25th IEEE International Conference on Image Processing. Athens, Ereece, Caye Daudt et al., 2018), FC-Siam-conc (Proceedings of the 2018 25th IEEE International Conference on Image Processing. Athens, Ereece, Caye Daudt et al. 2018.), UNet++MSOF (Remote Sensing, Peng et al., 2019), and IFN (Journal of Photogrammetry and Remote Sensing, Zhang et al., 2020). FC-EF uses an image-level fusion method to concatenate bi-temporal images into a single input to FCN. FC-Siam-diff and FC-Siam-conc both use a feature-level fusion method, using twin FCN to extract multi-level features, while FC-Siam-diff fuses bi-temporal information using feature differences and FC-Siam-diff fuses bi-temporal information using feature concatenation. UNet++_MSOF fuses global and fine-grained semantic features through a multi-faceted output fusion strategy in the UNet++ model, thereby combining change maps at different semantic levels to complete the change detection task. IFN multi-scale feature concatenation method, which applies channel attention and spatial attention to the concatenated bi-temporal features at all levels of the decoder. Deep supervision (i.e., calculating the supervision loss at each level of the decoder) is used to better train the intermediate layers.
[0109] (5) Experimental analysis and results:
[0110] Table 1 shows the comparative experimental results on the LEVIR-CD dataset.
[0111] Table 1 Comparative experimental results of change detection on the LEVIR-CD dataset
[0112]
[0113] As shown in Table 1, the accuracy (P) and recall (R) of the method in this example can reach 92.72% and 87.01% respectively on the LEVIR-CD dataset, the F1-score is 89.22%, and the intersection over union (IoU) is 81.02%. Compared with the classification results of FC-EF, FC-Siam-diff, FC-Siam-conc, UNet++_MSOF and IFN methods, the accuracy (P) is improved by about 1.75% to 7.24%, indicating that the introduction of Transformer to obtain rich global context information can effectively improve the model recognition change detection area. Comprehensive analysis of the results in Table 1 shows that the method proposed in this example can adaptively mine the context information of remote sensing feature maps from different angles, can fully and effectively perform accurate label prediction, and has good detection effects on objects of multiple scales.
[0114] In order to better describe the experimental results, the experimental data in Table 1 is used to complete the visual analysis result diagram ( Figure 7 , Figure 8 ).like Figure 7 As shown, the FC-EF, FC-Siam-diff, FC-Siam-conc, UNet++_MSOF and IFN methods will cause some buildings to be missed, as shown in the first and fourth rows of result diagrams. It can be seen that the method of this example can capture multi-level adaptive scale feature maps and achieve better building detection maps. The reason is that the REX module uses the spatial and channel correlation relationship between features to obtain more accurate features, making building change detection more accurate. At the same time, due to the introduction of the VIT module, appropriate contextual information is extracted for the object, and the feature representation capability is adaptively enhanced, which not only reduces incomplete and irregular semantic objects, but also better preserves geometric details and complex contours. As shown Figure 7 In the first, second, and fourth rows, the spatial location information is preserved. This shows that the proposed method has significantly improved the remote sensing image change detection.
[0115] The present embodiment proposes a remote sensing image change detection method based on semantic fusion, which combines the core advantages of Transformer and convolutional neural network. First, a deep encoder is designed to encode the local pixel blocks of the dual-phase remote sensing image, and at the same time, the Transformer module is introduced to further extract and integrate the global context association information of the image. In the decoding stage, upsampling technology is used to gradually restore the encoded high-level semantic features to the spatial resolution size of the original image. The decoder uses a jump connection mechanism to cascade multi-scale local detail features with global context features to achieve effective fusion of local and global semantic features. This fusion method can enhance the sensitivity and accuracy of change detection of various types of objects while maintaining accurate positioning of the change area, thereby improving the adaptability and robustness of the method in various scenarios.
[0116] Embodiment 2:
[0117] This embodiment proposes a remote sensing image change detection system based on semantic fusion, such as Fig. 9 As shown, it includes: an image acquisition module, a local feature extraction module, a global feature extraction module, an upsampling decoding module and an image classification module, and the image acquisition module, the local feature extraction module, the global feature extraction module, the upsampling decoding module and the image classification module are connected in sequence;
[0118] An image acquisition module, used to acquire multiple batches of first images to be detected and second images to be detected, wherein the first images to be detected and the second images to be detected are dual-phase images of the same area;
[0119] A local feature extraction module, used to input the first image to be detected and the second image to be detected into a depth encoder with a local interaction enhancement module, extract local features, and obtain a first local feature map and a second local feature map respectively;
[0120] A global feature extraction module, used for inputting the first local feature map and the second local feature map into a visual self-attention encoder respectively, extracting global features, and obtaining a global feature map;
[0121] An upsampling decoding module is used to input the global feature map into a cascade upsampling decoder, perform multiple upsampling and feature fusion operations, and obtain a feature map;
[0122] The image classification module is used to input the feature map into a classifier for classification to obtain a remote sensing image change detection map.
[0123] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0124] The protection scope of the present application is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.
Claims
1. A remote sensing image change detection method based on semantic fusion, characterized in that: include: Acquire multiple batches of first images to be detected and second images to be detected, wherein the first images to be detected and the second images to be detected are dual-phase images of the same area; Inputting the first image to be detected and the second image to be detected into a deep encoder with a local interaction enhancement module respectively, extracting local features, and obtaining a first local feature map and a second local feature map respectively; Inputting the first local feature map and the second local feature map into a visual self-attention encoder respectively, extracting global features, and obtaining a global feature map; The global feature map is input into a cascade upsampling decoder, and a feature map is obtained after multiple upsampling and feature fusion operations; The feature map is input into a classifier for classification to obtain a remote sensing image change detection map.
2. The remote sensing image change detection method based on semantic fusion according to claim 1 is characterized in that: The deep encoder with a local interaction enhancement module includes: a first level, a second level, a third level and a fourth level, and the local interaction enhancement module is arranged between the second level and the third level, between the third level and the fourth level and after the fourth level, the first level includes: a convolution layer and a pooling layer; the second level includes: a first residual module and 3 second residual modules; the third level includes: a first residual module and 4 second residual modules; the fourth level includes: a first residual module and 9 second residual modules.
3. The remote sensing image change detection method based on semantic fusion according to claim 2 is characterized in that: The local interaction enhancement module includes: a spatial interaction layer and a channel interaction layer. The spatial interaction layer is set between the second level and the third level, the channel interaction layer is set between the third level and the fourth level, and the spatial interaction layer is set after the fourth level. The spatial interaction layer is used to exchange features in the spatial dimension, and the channel interaction layer is used to exchange features in the channel dimension.
4. The remote sensing image change detection method based on semantic fusion according to claim 3 is characterized in that: The spatial interaction layer or channel interaction layer is calculated as follows: Among them, x0 and x1 represent the dual-phase characteristics, x 0 / 1 () is the spatial interaction layer, n is the number of batches, c is the number of channels of the first image to be detected or the second image to be detected, h is the height of the resolution, w is the width of the resolution, and M is the exchange mask, which consists of 1 and 0, 1 means exchange, and 0 means no exchange.
5. The remote sensing image change detection method based on semantic fusion according to claim 1 is characterized in that: The step of inputting the first local feature map and the second local feature map into a visual self-attention encoder respectively, extracting global features, and obtaining a global feature map comprises: Cascading the first local feature map and the second local feature map to obtain a cascade feature map; Expanding the cascade feature map into a two-dimensional image block, and linearly projecting the two-dimensional image block into a D-dimensional embedding space; Encoding the position information of the projected two-dimensional image block to obtain a coding feature; The encoded features are input into a visual self-attention encoder to obtain a global feature map. The visual self-attention encoder includes: a multi-head self-attention layer and a multi-layer perceptron layer. The multi-head self-attention layer calculates the attention value of each head respectively, and concatenates the attention values of all heads. The concatenated result is used as the output of the multi-head self-attention layer. The attention value of each head is calculated by performing an attention mechanism on the output of each Transformer layer. Each layer of the multi-layer perceptron layer includes a set of nonlinear activation functions.
6. The remote sensing image change detection method based on semantic fusion according to claim 5 is characterized in that: The calculation formula of the multi-head self-attention layer and the multi-layer perceptron layer is as follows: Z′ l =MSA(LN(Z l-1 ))+Z l-1 WITH l =MLP(LN(Z′ l ))+Z′ l Among them, Z l-1 is the encoded feature output by the l-1th Transformer layer, Z′ l is the feature output after calculation by the multi-head self-attention layer, Z l is the encoded feature output by the lth Transformer layer after calculation by the multi-layer perceptron layer, l∈{1,…,L}, L is the total number of layers of the visual self-attention encoder, and LN(·) is the layer normalization operation, which is calculated as: Among them, x is the vector representation of the neurons in the lth layer, g is the scaling parameter, b is the translation parameter, f is the activation parameter, μ is the mean of all neurons in the lth layer on the training sample, and σ is the variance of all neurons in the lth layer on the training sample.
7. The remote sensing image change detection method based on semantic fusion according to claim 5 is characterized in that: The multi-head self-attention layer calculates the attention value of each head respectively, and concatenates the attention values of all heads, and uses the concatenated result as the output of the multi-head self-attention layer. The attention value of each head is calculated by the attention mechanism of the output of each Transformer layer, and the calculation formula is as follows: MSA(Q,K,V)=Concat(head1,head2,…,head n )W 0 head i =Attention(Z l-1 IN i Q ,WITH l-1 IN i K ,WITH l-1 IN i V ) Among them, Z l-1 is the encoding feature output by the l-1th Transformer layer, is the query matrix, is the key matrix, is the value matrix, The parameter matrix of the multi-head self-attention layer, Concat() represents the cascade operation, head i is the attention value of the i-th self-attention head, e is the number of self-attention heads, and d k =d v =D / e, where D is the dimension of the D-dimensional embedding space.
8. The remote sensing image change detection method based on semantic fusion according to claim 1 is characterized in that: The step of inputting the global feature map into a cascade upsampling decoder, performing multiple upsampling and feature fusion operations to obtain a feature map includes: Inputting the global feature map into multiple upsampling layers for upsampling processing to obtain multiple upsampling features; the upsampling layer includes: 1 2x upsampling operator block and 2 convolution blocks, the convolution block includes: 3×3 convolution layer, the first batch of normalization layers and the first ReLU activation function, and the upsampling operator block uses bilinear interpolation calculation for upsampling; A skip connection is used to connect each multi-scale feature of the global feature map with each upsampled feature to obtain a feature map.
9. The remote sensing image change detection method based on semantic fusion according to claim 1 is characterized in that: The classifier includes: 2 1×1 convolutional layers, a second ReLU activation function and a second batch normalization layer.
10. A remote sensing image change detection system based on semantic fusion, characterized in that: include: An image acquisition module, used to acquire multiple batches of first images to be detected and second images to be detected, wherein the first images to be detected and the second images to be detected are dual-phase images of the same area; A local feature extraction module, used to input the first image to be detected and the second image to be detected into a depth encoder with a local interaction enhancement module, extract local features, and obtain a first local feature map and a second local feature map respectively; A global feature extraction module, used for inputting the first local feature map and the second local feature map into a visual self-attention encoder respectively, extracting global features, and obtaining a global feature map; An upsampling decoding module is used to input the global feature map into a cascade upsampling decoder, and obtain a feature map after multiple upsampling and feature fusion operations; The image classification module is used to input the feature map into a classifier for classification to obtain a remote sensing image change detection map.
Citation Information
Patent Citations
Transform and dense feature fusion-based remote sensing image change detection method and system
CN115690002A
Remote sensing image change detection network and detection method based on double twinborn branches
CN116524361A
Landslide recognition method based on laplacian pyramid remote sensing image fusion
US11521377B1
Cited By
Multispectral image change detection method and system
CN119295445A
A multispectral image change detection method and system
CN119295445B
Remote sensing image change detection method and system based on large model domain self-adaption
CN119625540A
A Remote Sensing Image Change Detection Method and System Based on Large Model Domain Adaptation
CN119625540B
Hierarchical Transform attention feature fusion-based iPSCs set pluripotency evaluation method
CN119672711A
