Neural network-based remote sensing image change detection method and system

Abstract

The disclosure provides a neural network-based remote sensing image change detection method and system, comprising: acquiring first and second remote sensing images of different time phases; using a Transformer encoder based on a multi-head jump attention mechanism to extract global spatial features and image block spatial features of the first and second remote sensing images at different scales, respectively, to obtain first and second remote sensing image fusion features; wherein the multi-head jump attention mechanism is specifically: for the self-attention module of the encoder, the jump attention mechanism calculation results of the global space and the image block space are obtained based on the difference between the first and second remote sensing images; performing decoding operation after fusing the first and second remote sensing image fusion features at different scales to obtain a multi-scale feature map; based on the multi-scale feature map, using a preset classifier to obtain a change map, realizing change detection of remote sensing images.

Description

Technical Field

[0001] This disclosure belongs to the field of remote sensing image building change detection technology, and particularly relates to a method and system for remote sensing image change detection based on neural networks. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Change detection primarily refers to changes noticed for the first time in a visual process. In change detection tasks, if a computer is to perform this process, it generally needs to obtain feature representations of the same location at different times. Therefore, the quality of the acquired image has a significant impact on the detection results. Recently, with the improvement of the spectral resolution of launched satellites, the accuracy of remote sensing images has also been greatly improved, providing more refined scale representations for tasks such as terrain detection and urban change detection. Since the goal of change detection is to find changes in objects of interest in an image, models are commonly used to detect changes in urban buildings, surface water, forest cover, water resources, urban scene classification, and in the medical field. Depending on the application scenario, different target changes need to be considered. A good change detection model should be able to avoid irrelevant changes caused by atmospheric changes, seasonal changes, and changes in light intensity, and be able to effectively extract feature information from high-resolution images to reconstruct the approximate outline of the target.

[0004] According to the inventors, change detection tasks initially relied on traditional methods such as post-classification comparison (PCC), principal component analysis (PCA), and change vector analysis (CVA) for feature extraction. While these methods met the requirements to some extent, they were prone to misclassification, resulting in low detection accuracy, and the training samples depended on human experience. Subsequently, change detection models based on convolutional neural network (CNN) architectures emerged, improving multi-scale feature extraction and generalization capabilities. Further research has yielded significant results for CNN-based models, but their limitations have also become apparent. The convolutional operation, due to its limited receptive field, reduces the performance of long-term semantic information representation, which plays a crucial role in change detection. To address this issue, researchers have begun to study the Transformer architecture. The Transformer is a novel network structure capable of extracting broader semantic information. The introduction of the Transformer allows change detection tasks to more fully utilize global features in images.

[0005] While these methods can handle change tasks with greater efficiency and accuracy, they still have the following drawbacks: changes in remote sensing images are usually small, and current convolutional or Transformer-based methods struggle to obtain clear change maps of small buildings; simply using Transformer for feature extraction does not fully utilize the richer semantic information in bitemporal images, as bitemporal images contain more information in the time dimension than single images. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure provides a method and system for detecting changes in remote sensing images based on neural networks. The proposed method, based on a jump attention calculation approach, effectively improves the accuracy of detection results and offers new solutions for related needs, significantly enhancing the accuracy and efficiency of management tasks such as urban change assessment or post-disaster damage assessment.

[0007] According to a first aspect of the present disclosure, a method for detecting changes in remote sensing images based on a neural network is provided, comprising:

[0008] Acquire first and second remote sensing images at different time points;

[0009] A Transformer encoder based on a multi-head skip attention mechanism is used to extract global spatial features and image patch spatial features of the first and second remote sensing images at different scales to obtain the first and second remote sensing image fusion features. Specifically, the multi-head skip attention mechanism is as follows: for the self-attention module of the encoder, the calculation results of the skip attention mechanism in the global space and image patch space are obtained based on the differences between the first and second remote sensing images.

[0010] After fusing the fusion features of the first remote sensing image and the fusion features of the second remote sensing image at different scales, a decoding operation is performed to obtain a multi-scale feature map;

[0011] Based on the multi-scale feature map, a change map is obtained using a preset classifier to achieve change detection in remote sensing images.

[0012] Furthermore, the Transformer encoder includes four channels, with the first remote sensing image and the second remote sensing image each corresponding to two channels; for each remote sensing image, one channel is used to extract features of the first and second remote sensing images in the global space using the Transformer encoder, and the other channel is used to extract features of the first and second remote sensing images in the image patch space using the Transformer encoder.

[0013] Furthermore, the extraction of the spatial features of the image blocks requires dividing the input image into several non-overlapping image blocks, and then feeding them into a Transformer encoder based on a multi-head jump attention mechanism.

[0014] Furthermore, the fusion of the first remote sensing image fusion feature and the second remote sensing image fusion feature at different scales specifically involves: for each scale, calculating the difference between the two images based on the first remote sensing image fusion feature and the second remote sensing image fusion feature; merging the difference with the first remote sensing image fusion feature and the second remote sensing image fusion feature to obtain preliminary fusion features; and merging the preliminary fusion features at each scale in ascending order of scale to obtain the final fusion features.

[0015] Furthermore, the decoder employs a multilayer perceptron to aggregate the difference features and original features of the first and second remote sensing images.

[0016] Furthermore, the step of obtaining a change map based on the multi-scale feature map using a preset classifier specifically involves: classifying regions with changes and regions without changes based on the multi-scale feature map using a preset classifier, and representing the classification results in a binary format to obtain the change map.

[0017] According to a second aspect of the present disclosure, a remote sensing image change detection system based on a neural network is provided, comprising:

[0018] The data acquisition unit is used to acquire first and second remote sensing images at different time phases, respectively;

[0019] The encoding unit is used to extract global spatial features and image patch spatial features of the first remote sensing image and the second remote sensing image at different scales using a Transformer encoder based on a multi-head skip attention mechanism, to obtain the first remote sensing image fusion features and the second remote sensing image fusion features; wherein, the multi-head skip attention mechanism is specifically: for the self-attention module of the encoder, the calculation results of the skip attention mechanism of the global space and the image patch space are obtained based on the difference between the first remote sensing image and the second remote sensing image;

[0020] The decoding unit is used to fuse the fusion features of the first remote sensing image and the fusion features of the second remote sensing image at different scales and then perform a decoding operation to obtain a multi-scale feature map.

[0021] The detection unit is used to obtain a change map based on the multi-scale feature map using a preset classifier, thereby realizing change detection in remote sensing images.

[0022] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor executes the program to implement the aforementioned method for detecting changes in remote sensing images based on a neural network.

[0023] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the aforementioned method for detecting changes in remote sensing images based on a neural network.

[0024] Compared with the prior art, the beneficial effects of this disclosure are:

[0025] (1) The solution described in this disclosure provides a method and system for detecting changes in remote sensing images based on neural networks. The solution is based on the proposed jump attention calculation method, which can effectively improve the accuracy of the detection results and provide new solutions for related needs, thereby greatly improving the accuracy and efficiency of management work such as urban change assessment or post-disaster damage assessment.

[0026] (2) The scheme described in this disclosure redesigns the calculation method of Q, K, V in the self-attention module by mimicking the human action of finding differences between two images, which effectively enhances the ability of Transformer to extract features in dual-phase images. In the scheme, each phase of the image corresponds to two channels in the encoder. One channel uses Transformer to focus on global information, and the other channel divides the image into a set of non-overlapping image blocks. Then, Transformer is used to capture the feature information of the same area in the two images. The feature information extracted by the dual channels in the two spatial regions can not only retain the advantages of Transformer in long-distance feature extraction, but also improve the ability to further improve the extraction of detailed features.

[0027] (3) In order to further improve the model decoding capability, the scheme designs a progressive decoder with an MLP (Multilayer Perceptron) layer based on the U-net decoder, which further improves the detection accuracy of the model.

[0028] Advantages of this disclosure in additional aspects will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0029] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0030] Figure 1This is a flowchart illustrating the overall implementation of the solution described in the embodiments of this disclosure;

[0031] Figure 2 This is a schematic diagram of the overall network structure of the neural network-based remote sensing image change detection method described in the embodiments of this disclosure;

[0032] Figure 3 This is a schematic diagram of the dual-channel feature extraction structure described in the embodiments of this disclosure;

[0033] Figure 4 This is a schematic diagram of the image patch spatial skip attention mechanism described in the embodiments of this disclosure;

[0034] Figure 5 This is a schematic diagram of the overall structure of the jump attention module described in the embodiments of this disclosure;

[0035] Figure 6 This is a schematic diagram of the LEVIR-CD dataset comparison experiment described in the embodiments of this disclosure;

[0036] Figure 7 This is a schematic diagram of the comparative experiment of the DSIFN-CD dataset described in the embodiments of this disclosure;

[0037] Figure 8 This is a diagram showing the experimental results of network structure ablation as described in the embodiments of this disclosure. Detailed Implementation

[0038] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0041] Where there is no conflict, the embodiments and features described herein can be combined with each other.

[0042] Example 1:

[0043] The purpose of this embodiment is to provide a method for detecting changes in remote sensing images based on neural networks.

[0044] The shortcomings of the background technology are mainly caused by the characteristics of the change detection task itself, specifically including: (1) The change detection task of buildings needs to identify the changed area based on the difference between two images and only focuses on the change of buildings. Therefore, the changes of other ground objects will affect the performance of the building change detection task. (2) The model needs to extract subtle global and local features, but in this process, it is easy to ignore the features of smaller buildings, which directly affects the detection accuracy. (3) When humans look for differences in two images, they will purposefully look for differences in the same location. This efficient method is not reflected in the current change detection method. (4) The current method extracts feature information from two temporal images separately, but we have found that extracting features from two temporal images at the same time can make better use of the feature information in the image.

[0045] To address the aforementioned problems, this embodiment proposes a method for detecting changes in remote sensing images based on neural networks, comprising:

[0046] Acquire first and second remote sensing images at different time points;

[0047] A Transformer encoder based on a multi-head skip attention mechanism is used to extract global spatial features and image patch spatial features of the first and second remote sensing images at different scales to obtain the first and second remote sensing image fusion features. Specifically, the multi-head skip attention mechanism is as follows: for the self-attention module of the encoder, the calculation results of the skip attention mechanism in the global space and image patch space are obtained based on the differences between the first and second remote sensing images.

[0048] After fusing the fusion features of the first remote sensing image and the fusion features of the second remote sensing image at different scales, a decoding operation is performed to obtain a multi-scale feature map;

[0049] Based on the multi-scale feature map, a change map is obtained using a preset classifier to achieve change detection in remote sensing images.

[0050] In a specific implementation, the Transformer encoder includes four channels, with the first remote sensing image and the second remote sensing image each corresponding to two channels. For each remote sensing image, one channel is used to extract features of the first and second remote sensing images in the global space using the Transformer encoder, and the other channel is used to extract features of the first and second remote sensing images in the image patch space using the Transformer encoder.

[0051] In a specific implementation, the extraction of the spatial features of the image blocks requires dividing the input image into several non-overlapping image blocks, and then feeding them into a Transformer encoder based on a multi-head jump attention mechanism.

[0052] In specific implementation, the fusion of the first remote sensing image fusion feature and the second remote sensing image fusion feature at different scales is specifically as follows: for each scale, the difference between the two images is calculated based on the first remote sensing image fusion feature and the second remote sensing image fusion feature; the difference is merged with the first remote sensing image fusion feature and the second remote sensing image fusion feature to obtain preliminary fusion features; the preliminary fusion features at each scale are merged step by step in order from smallest to largest scale to obtain the final fusion features.

[0053] To facilitate understanding, the solution described in this embodiment will be explained in detail below with specific examples:

[0054] In the specific implementation, samples from the public datasets LEVI R-CD and DSI FN-CD are used to illustrate the scheme described in this embodiment. These two datasets contain 637 pairs of images with a resolution of 1024×1024 and 3988 pairs with a resolution of 512×512, respectively. Since directly inputting the original images into the network for detection is too slow, all images are segmented into non-overlapping image blocks with a resolution of 256×256 to reduce the computational load per batch. During model training, these blocks are divided into training, validation, and test sets based on the original sample classification results.

[0055] In this embodiment, a four-channel Transformer structure is fused into a dual-temporal twin change detection channel. This is because we want to retain the advantages of the Transformer in long-range feature extraction while compensating for its insufficient ability to extract fine features. The overall network structure is as follows: Figure 2 As shown, this network structure takes dual-temporal high-resolution images as input, with each temporal image fed into two channels. One channel uses a skip-attention Transformer module to extract contextual semantic information globally from the image of the other temporal phase. The other channel first segments the input image into multiple non-overlapping blocks, then feeds these blocks into the skip-attention Transformer module for feature extraction from the corresponding temporal image of the other channel. The two skip-attention Transformer modules employ different computational methods. The main reason for this design is that the Transformer has a strong advantage in global contextual semantic extraction, but its performance in local feature extraction is not as good as convolutional operations. Experiments have shown that the design details of this example can retain the advantages of the Transformer and improve the ability to extract fine features.

[0056] Specifically, the process of feature extraction and overlay in global space and image patch space is as follows: Figure 3 As shown, specifically, the resolution of the input image pair is H×W×3, where H and W are the height and width of the input image, respectively, and the channel dimension is 3.

[0057] (1) Feature extraction based on global space:

[0058] The input image is first fed into a downsampling layer to resize the image, and then enters a multi-head over-attention (MHOA) Transformer encoder for global spatial contextual semantic extraction. The output feature map after feature extraction has a resolution of [missing information]. Where i∈{1,2,3,4},C i+1 >C i The feature map output of each level serves as the input to the next level module, where C represents the number of channels in the image. For example, if the original image is an RGB image, then C equals 3.

[0059] (2) Feature extraction based on image patch space:

[0060] This channel focuses only on variations within the same region of the image. First, patch embedding is performed on the image, dividing it into multiple non-overlapping patches. Then, an MHOA encoder is used to establish semantic relationships within these patches, thus obtaining detailed feature information at each level of the image patch space. The feature map output by this channel has a resolution of [resolution value missing]. in Here, t is used to distinguish it from features extracted based on the global space. A logical diagram of feature extraction in image patch space is shown below. Figure 4 As shown.

[0061] The key structure for the above feature extraction is the MHOA encoder module. The structure of the skip attention module in the encoder is as follows: Figure 5 As shown, q1, k1, and v1 are calculated from the image at time T1, and q2, k2, and v2 are calculated from the image at time T2; the calculation process is the same. This module is used to extract features from the two time-phase images separately. Unlike other methods that extract features only from one time phase, ignoring the temporal information in the dual-time-phase images, we utilize both time-phase images simultaneously. While extracting features from the first time phase, we combine the feature information from the same location at the second time phase to improve the feature extraction capability of the first time phase image. The principle for feature extraction from the second time phase image is the same. This process is similar to the human process of finding differences between two images. Experimental results show that... Figure 5The method can utilize the information in the time dimension of dual-phase images, which has a stronger ability to express image features and can greatly improve detection accuracy.

[0062] Because the Transformer has a larger receptive field, it can fully utilize long-term spatiotemporal semantic information to generate richer feature representations. In the global-based space, each channel includes a downsampling layer and a sequence reduction process; in the image patch-based space, the image patch is directly sent to the MHOA module after the sequence reduction process, without the need for downsampling. In MHOA, the input features are mapped to three high-dimensional spaces: Query(Q), Key(K), and Value(V).

[0063] Q p =T p W q

[0064] K p =T p W k

[0065] V p =T p W v

[0066] in, W represents the vector expansion of the input feature image. q W k , There are three learnable linear mapping layers, where d is the dimension of the input channels. The proposed skip attention mechanism based on global space and image patch space is computed as follows:

[0067] Global space:

[0068] Image patch space:

[0069] Among them, Q,K, Representing Query, Key, Value, p and This indicates the preceding or following time phase. If p represents the preceding time phase, then... ⊙ represents the post, and vice versa. ⊙ represents the Hadamard product, which is the multiplication of matrices at the same positions. During encoding, Q, K, and V have the same dimension. A schematic diagram of the image patch space skip attention mechanism is shown below. Figure 3 As shown.

[0070] Preferably, to reduce the computational complexity of the equations, the sequence reduction process is as follows:

[0071]

[0072]

[0073] Where S is the sequence to be reduced, and Reshape() and Linear() represent tensor deformation and linear layer operations, respectively. To improve feature representation, R is the reduction rate, which is a hyperparameter; in this embodiment, R = 4 is used.

[0074] MHOA also includes an MLP layer and a Norm layer. The calculation formula for the entire module is as follows:

[0075]

[0076]

[0077] Where SR represents the sequence reduction process. This represents the fusion feature output of MHOA, and LN stands for Layer Norm.

[0078] Then, the feature maps obtained in the global space and the image patch space are superimposed. The purpose of this is to focus on changes in local detail features and leverage the advantages of Transformer's long-range contextual semantic extraction. The feature resolution size extracted based on the image patch space is... Its resolution is inconsistent with that of features extracted from the global space at the same level. Therefore, it is necessary to perform upsampling or downsampling operations to adjust its size so that it can be consistent with the size of features extracted from the global space. Then, it is superimposed to make the extracted feature information richer.

[0079] After extracting richer semantic information, it is fed into the difference module to calculate the differences, and then passes through a multi-scale feature fusion module to reduce feature loss during the calculation process. For the difference module: the difference module of the i-th layer first processes the feature map F extracted by the encoder of the i-th layer. i p The features of p∈{pre,post} are merged, and then features from different time domains are mapped to the same time domain. The optimal distance metric at this scale is calculated, and this calculation is learnable during training. Finally, the difference features are mapped back to the original spatial domain. The entire computation process of this module is shown below:

[0080]

[0081] After obtaining the differential features, they are merged with the output features and then fused through an MLP layer. The fusion process at each layer is as follows:

[0082]

[0083] Among them, Cemb This indicates the number of embedding channels.

[0084] Inspired by the U-net decoder, this example designs a decoder consisting of simple MLP layers to enhance features extracted from the MHOA module and reconstruct pixel-level output. Specifically, it utilizes four enhanced effective feature layers for feature fusion, and then maps low-dimensional features to high-dimensional feature layers through upsampling. A convolutional connection layer is added after the upsampling layer to avoid artifacts caused by upsampling. The final result is a feature map with the same height and width as the input image. Unlike the U-net decoder, this example uses a simple MLP layer to aggregate differential and original features because research shows that simple MLP layers also have the ability to model long-term dependencies. The main parts of the decoder are as follows:

[0085]

[0086] In the final change classification prediction part, this example uses two upsampling layers to fuse the final multi-scale feature map. Upsampled to the same resolution H×W as the input image, finally generating H×W×N. cls ChangeMap, where N cls =2 represents the number of classes, i.e., changed and unchanged. Based on the multi-scale feature map, a preset classifier is used to classify regions that have changed and those that have not, and the classification results are binarized to obtain a change map. This process can be represented as follows:

[0087]

[0088] This invention uses the F1 score for the variation category as the primary evaluation metric. The F1 score is calculated based on the test's precision and recall:

[0089]

[0090] In addition, recall, precision, intersection over union (IoU) of the change category, and overall accuracy (OA) are shown below:

[0091]

[0092]

[0093]

[0094]

[0095] Where TP, TN, FP, and FP represent true samples, true negative samples, false positive samples, and false negative samples, respectively.

[0096] In this implementation, the model was built and trained in PyTorch using an NVIDIA RTX2080 GPU. The initial learning rate was 0.0001, and the number of epochs was set to 200. The batch size for training the model was set to 8. The network was optimized using the AdamW optimizer with training weight decay of 0.01 and a beta value of (0.9, 0.999). The Transformer had 8 MHOA heads. This example also trained seven other change detection network models, including FC-EF (Fully Convolutional Early Fusion), FC-Siam-Di, FC-Siam-Conc, Changeformer, BIT (Binary Neural Networks), SNUnet, and DTCDSCN (Dual Task Constrained Deep Siamese Convolutional Network). The performance of these seven network models on the two datasets and the evaluation parameter values ​​are compared with the network model proposed in this example. Figure 6 and Figure 7 As shown.

[0097] like Figure 6 As shown, (a) image at time T1; (b) image at time T2; (c) labeled image; (d) FE-EF; (e) FC-Siam-Di; (f) FC-Siam-Conc; (g) ChangeFormer; (h) BIT; (i) SNUNet; (j) DTCDSCN; (k) SOAT-UNet (which corresponds to the scheme described in this embodiment);

[0098] like Figure 7 As shown, (a) image at time T1; (b) image at time T2; (c) labeled image; (d) FE-EF; (e) FC-Siam-Di; (f) FC-Siam-Conc; (g) ChangeFormer; (h) BIT; (i) SNUNet; (j) DTCDSCN; (k) SOAT-UNet (corresponding to the scheme described in this example);

[0099] It can be seen that BIT, DTCDSCN, SNUnet, and ChangeFormer achieve better results than CNN-based models such as FC-EF, FC-Siam-Di, and FC-Siam-Conc. This is attributed to the attention mechanism's ability to extract long-range semantic information. DTCDSCN achieves good results through a dual attention module; ChangeFormer uses a Transformer-based encoder, which is an effective attempt to extract features using a pure Transformer encoder, but this model only focuses on the long-term spatiotemporal correlation of images, ignoring local details. This demonstrates that the refinement of feature extraction directly affects the final detection results. In this example, compared to the best-performing comparison method on the LEVIR-CD dataset, the proposed model improves F1, IoU, accuracy, and recall by 0.6%, 1.03%, 0.84%, and 0.3%, respectively; and compared to the best-performing model on the DSIFN-CD dataset, it improves F1, IoU, accuracy, and recall by 2.14%, 3.53%, 1.68%, and 2.58%, respectively. The results show that the method presented in this example (SOAT-Unet) outperforms other methods in terms of change detection accuracy and has stronger generalization ability on different datasets.

[0100] To validate the contributions of the proposed Transformer-based skip attention encoder and multi-scale fusion decoder, we conducted ablation experiments on the LEVIR-CD dataset. This section includes seven ablation experiments. The results are as follows: Figure 8 As shown in the diagram. The first group uses ChangeFormer as the base network and employs a Multi-Head Self-Attention (MHSA) module to extract feature information in the global space. The second group replaces the MHSA module of the first group with a Multi-Head Skip Attention (MHOA) module. The third group uses MHOA as the encoder in the image patch space. The fourth group, based on the first group, is the decoder (MFFD) composed of MLP layers proposed in this example. The fifth group replaces the MHSA encoder of the fourth group with an MHOA encoder in the global space. The sixth group replaces the global space MHOA encoder of the fifth group with an MHOA encoder in the image patch space. The seventh group combines the global space and image patch space MHOA encoders and utilizes MFFD to enhance decoding performance. Experimental results are shown in the diagram. Figure 8 As shown. By comparing the results of the first and second groups, it can be seen that global spatial MHOA can improve the change detection effect. The third group improves the encoder's ability to extract features of smaller buildings, thereby further improving parameters such as F1. By comparing the results of the fourth, fifth, and sixth groups with those of the first, second, and third groups, it can be concluded that the MFFD decoder can further improve the model's detection performance on the original basis.

[0101] Building upon the fifth group, the final group combines the advantages of global and image patch space MHOA encoders, further improving the ability to extract fine features and showing significant improvements across all evaluation metrics. F1, accuracy, IoU, recall, and OA increased by 1.48%, 1.78%, 2.63%, 1.17%, and 0.9%, respectively.

[0102] Furthermore, in specific implementation, such as Figure 1 As shown, the specific implementation of the solution described in this embodiment includes the following steps:

[0103] S1. Cropping and segmenting the remote sensing image and the labeled image to divide the image into non-overlapping image blocks, each with a pixel size of 256×256.

[0104] S2. Divide the preprocessed remote sensing images and labeled images into training set, validation set and test set.

[0105] S3. The proposed improved Transformer structure using the jump attention module is used as the encoder to simultaneously extract features from the bi-temporal images in both the image patch space and the global space. Then, a change detection network model is built by combining this with the improved U-net decoder. The Siamese jump attention network model is trained using the training set for a total of 200 iterations.

[0106] S4. After each iteration during training, the validation set obtained in S2 is used to evaluate the trained model. The main evaluation criterion is the size of the F1 score. After 200 iterations, the weight file with the highest F1 score is retained.

[0107] S5. Using the test set images obtained in S2 as input images, the final detection model is used to detect the changes in the buildings.

[0108] Example 2:

[0109] The purpose of this embodiment is to provide a remote sensing image change detection system based on neural networks.

[0110] A method for detecting changes in remote sensing images based on neural networks, comprising:

[0111] The data acquisition unit is used to acquire first and second remote sensing images at different time phases, respectively;

[0112] The encoding unit is used to extract global spatial features and image patch spatial features of the first remote sensing image and the second remote sensing image at different scales using a Transformer encoder based on a multi-head skip attention mechanism, to obtain the first remote sensing image fusion features and the second remote sensing image fusion features; wherein, the multi-head skip attention mechanism is specifically: for the self-attention module of the encoder, the calculation results of the skip attention mechanism of the global space and the image patch space are obtained based on the difference between the first remote sensing image and the second remote sensing image;

[0113] The decoding unit is used to fuse the fusion features of the first remote sensing image and the fusion features of the second remote sensing image at different scales and then perform a decoding operation to obtain a multi-scale feature map.

[0114] The detection unit is used to obtain a change map based on the multi-scale feature map using a preset classifier, thereby realizing change detection in remote sensing images.

[0115] Furthermore, the system described in this embodiment corresponds to the method described in Embodiment 1, and its technical details have been described in detail in Embodiment 1, so they will not be repeated here.

[0116] In further embodiments, the following is also provided:

[0117] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0118] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0119] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0120] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0121] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0122] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0123] The above embodiments provide a method and system for detecting changes in remote sensing images based on neural networks, which can be implemented and has broad application prospects.

[0124] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A neural network-based remote sensing image change detection method, characterized in that, The method comprises the following steps: acquiring first remote sensing images and second remote sensing images at different time phases; using a Transformer encoder based on a multi-head skip attention mechanism to respectively extract global spatial features and image block spatial features of the first remote sensing images and the second remote sensing images at different scales, and obtain first remote sensing image fusion features and second remote sensing image fusion features; wherein the multi-head skip attention mechanism is specifically as follows: for the self-attention module of the encoder, the calculation results of the global spatial and image block spatial skip attention mechanisms are obtained based on the differences between the first remote sensing images and the second remote sensing images; performing decoding operation on the first remote sensing image fusion features and the second remote sensing image fusion features at different scales after fusion, and obtaining a multi-scale feature map; based on the multi-scale feature map, using a preset classifier to obtain a change map to realize change detection of remote sensing images; the Transformer encoder comprises four channels, and the first remote sensing images and the second remote sensing images correspond to two channels respectively; for the two channels corresponding to each remote sensing image, one channel is used to extract the features of the first remote sensing images and the second remote sensing images in the global space by using the Transformer encoder, and the other channel is used to extract the features of the first remote sensing images and the second remote sensing images in the image block space by using the Transformer encoder; the calculation of the global spatial and image block spatial skip attention mechanisms is specifically as follows: Global space: Image block space: wherein, Query, Key, Value, respectively, p and denotes the previous or the next time phase, denotes the Hadamard product, i.e. the multiplication of the matrices in the same position, N being the dimension of the input vector.

2. The neural network-based remote sensing image change detection method of claim 1, wherein, the extraction of the image block spatial features needs to divide the input image into a plurality of non-overlapping image blocks, and then feed them into the Transformer encoder based on the multi-head skip attention mechanism.

3. The neural network-based remote sensing image change detection method of claim 1, wherein, the fusion of the first remote sensing image fusion features and the second remote sensing image fusion features at different scales is specifically as follows: for each scale, the differences between the two images are calculated based on the first remote sensing image fusion features and the second remote sensing image fusion features; the differences and the first remote sensing image fusion features and the second remote sensing image fusion features are merged to obtain preliminary fusion features; the preliminary fusion features at each scale are merged in order of scale from small to large to obtain the final fusion features.

4. The neural network-based remote sensing image change detection method of claim 1, wherein, the decoder uses a multi-layer perception to aggregate the difference features and the original features of the first remote sensing images and the second remote sensing images.

5. The neural network-based remote sensing image change detection method of claim 1, wherein, based on the multi-scale feature map, using a preset classifier to obtain a change map is specifically as follows: based on the multi-scale feature map, using a preset classifier to classify the regions with changes and the regions without changes, and performing binaryzation on the classification results to obtain a change map.

6. A neural network-based remote sensing image change detection system, characterized in that, The method comprises the following steps: a data acquisition unit is configured to acquire first remote sensing images and second remote sensing images at different time phases respectively; The encoding unit is configured to extract global spatial features and image block spatial features of the first remote sensing image and the second remote sensing image at different scales by using a Transformer encoder based on a multi-head jump attention mechanism, to obtain first remote sensing image fusion features and second remote sensing image fusion features; wherein the multi-head jump attention mechanism is specifically as follows: for a self-attention module of the encoder, a jump attention mechanism calculation result of global space and image block space is obtained based on a difference between the first remote sensing image and the second remote sensing image; The decoding unit is configured to perform decoding operation on the first remote sensing image fusion features and the second remote sensing image fusion features at different scales after fusion, to obtain a multi-scale feature map; The detection unit is configured to obtain a change map by using a preset classifier based on the multi-scale feature map, to realize change detection of the remote sensing image. The Transformer encoder includes four channels, and the first remote sensing image and the second remote sensing image correspond to two channels respectively; for the two channels corresponding to each remote sensing image, one channel is configured to extract features of the first remote sensing image and the second remote sensing image in global space by using the Transformer encoder, and the other channel is configured to extract features of the first remote sensing image and the second remote sensing image in image block space by using the Transformer encoder. The jump attention mechanism calculation of the global space and the image block space is specifically as follows: Global space: Image block space: wherein, Query, Key, Value, p and denotes the previous or the next time phase, denotes the Hadamard product, i.e. the multiplication of the matrices in the same position, N being the dimension of the input vector.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory to run on the processor, characterized in that, The processor executes the program to realize the neural network-based remote sensing image change detection method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the neural network-based remote sensing image change detection method of any one of claims 1-5.

Images