A remote sensing image change detection method and system based on a Transformer and graph semantic guidance

Abstract

The application discloses an image processing technical field and relates to a remote sensing image change detection method and system based on a Transformer and graph semantic guidance. In the overall framework, in addition to connecting CNN and the Transformer in multiple layers in series and in parallel, a novel graph semantic module (GSM) is constructed to guide the network to perform more detailed feature extraction through graph semantic features. In the module, each channel of a feature map is regarded as a feature vertex, the relationship between the feature vertices is represented by an adjacency matrix, and the multi-hop neighbor aggregation is used to enhance the model's ability to capture long-distance dependencies, so that graph semantic features with richer information are obtained. The graph semantic features are used to guide the backbone features extracted by the network in a subsequent semantic fusion Transformer module (SFTM), and the two are adaptively updated. The SFTM mainly comprises cross attention for completing information exchange, a local information compensation layer for supplementing local information loss, and a reinforcement feedforward network for optimizing input. The method can improve the performance of the network and improve the accuracy of change detection.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and system for detecting changes in remote sensing images based on Transformer and graph semantic guidance. Background Technology

[0002] Change detection in remote sensing images involves quantitatively analyzing and determining the characteristics and processes of surface changes from remote sensing images covering the same geographic area at different times. High-resolution remote sensing images, due to varying imaging conditions and the complex and diverse nature of objects within the scene, often exhibit different spectral characteristics at different times and spatial locations for objects with the same semantic concept. This presents a significant challenge for change detection. To accurately identify changed regions in complex scenes, change detection models need to recognize high-level semantic information about these regions, distinguishing genuine changes from complex, seemingly unrelated areas of change.

[0003] By utilizing self-attention to model the semantic relationships between pixel pairs in spatiotemporal space, better detection results can be achieved. In 2020, Hao Chen et al. from Beihang University proposed STANet, a novel Siamese network based on metric and spatiotemporal attention. It divides the network into different scales, embeds self-attention within each small region to model long-distance dependencies between pixels, and uses a batch-balanced contrastive loss function to alleviate the problem of sample imbalance, helping the network extract features rich in contextual information. However, the model has very low computational efficiency, high computational complexity, and its computational complexity increases quadratically with the number of pixels.

[0004] Many current change detection methods based on deep convolutional neural networks (CNNs) fail to effectively capture long-distance dependencies between pixels. Against this backdrop, Hao Chen et al. from Beijing University of Aeronautics and Astronautics proposed the Bitemporal Image Transformer (BIT) detection algorithm, which achieved good results in change detection tasks. However, because each head of the multi-head attention in the Transformer is only responsible for a subset of the input token sequence, when the channel dimension of the subset is small, the dot product operation of the query and key cannot form an information function, leading to a decrease in network performance. Furthermore, BIT's direct subtraction feature fusion method not only easily damages the feature structure but may also introduce noise interference. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a remote sensing image change detection method and system based on Transformer and graph semantic guidance, so that the final change detection results of dual-temporal remote sensing images are more accurate.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a remote sensing image change detection method based on Transformer and graph semantic guidance, comprising:

[0007] a. Perform image registration on the remote sensing images to obtain preprocessed dual-time remote sensing images, and then transmit these images into the network proposed in this invention.

[0008] b. In the first two layers of the network encoding stage, two convolutional layers are used for shallow feature extraction. In the last three layers of the network encoding stage, a Semantic Fusion Transformer Module (SFTM) is used for deep feature extraction. Before deep feature extraction, a Graph Semantics Module (GSM) is used to generate graph semantic features to guide the backbone features for more detailed feature extraction. When the spatial resolution of the image is reduced to one-quarter of the original image, the backbone features and graph semantic features of the dual-image are concatenated separately for channel adjustment. The fused features are then used by two Semantic Fusion Transformer Modules (SFTM) to complete the encoding part of the entire network.

[0009] c. In the decoding part of the network, the change information is gradually recovered through two semantic fusion Transformer modules (SFTM) and two convolutional layers. The upsampling operation includes a scale fusion-A (SF-A) module and a scale fusion-B (SF-B) module. Finally, a 1×1 convolution adjusts the number of channels of the image features to two, thus obtaining a semantic binary change map.

[0010] d. Throughout the training process, a hybrid loss function based on Dice and cross-entropy loss was used to mitigate the impact of imbalanced samples. By updating the loss function through gradients, the optimal change detection model can be obtained after multiple rounds of training.

[0011] The image registration in step a includes geometric coarse correction and geometric fine correction.

[0012] The GSM in step b primarily draws upon the theory of graph convolutional networks. In GSM, each channel of a feature is treated as a feature vertex to achieve non-local operations on the features. Because the feature distributions differ across layers, the relationships between features are not easily obtained, so the adjacency matrix A cannot be manually set. The core idea of ​​this method can be simply expressed by the following equation:

[0013] X out =WX in (A1×A2×A3+A4) (4)

[0014] The core of this formula lies in how to model the relationships between feature vertices using an adjacency matrix. This matrix can be divided into four parts: A1, A2, A3, and A4, described in detail below:

[0015] 1) A1 represents an N×N identity matrix. Since the adjacency relationships between each feature vertex are unknown, A1 is used to represent the feature vertex itself.

[0016] 2) A2 represents an N×N diagonal matrix, which is the weight of each feature vertex calculated during the first neighborhood aggregation process. Specifically, the input tensor first passes through a 1×1 one-dimensional convolutional layer and is normalized to between 0 and 1. These values ​​are then arranged diagonally to serve as the weights for each feature vertex. The formula for calculating A2 is as follows:

[0017] A2 = T(softmax(WX) in (5)

[0018] Where W represents the weights of a 1×1 convolutional layer, and T represents arranging the results diagonally.

[0019] 3) Similar to A2, A3 also represents an N×N diagonal matrix, which is the weight of each feature vertex calculated during the second neighborhood aggregation process. The calculation process of A3 is as follows:

[0020] A3 = T(softmax(W(X)) in (A2)))) (6)

[0021] 4) A4 is an N×N adjacency matrix. It is optimized through backpropagation to generate relationships between any two feature vertices during training. A4 is initialized with small initial values ​​to ensure the stability of initial learning, and its element values ​​are not numerically restricted. Therefore, A4 can flexibly learn global information while customizing the dependencies between various feature vertices contained in different layers.

[0022] Since the elements in the adjacency matrix can represent not only the existence or absence of connections between nodes, but also the strength of those connections, and even represent unknown relationships, these representations can be optimized through adaptive learning to form new adjacency matrices. These adjacency matrices can be used to capture the spatial connections and changes between ground features. In this framework, feature vertices are updated using the adjacency matrices obtained through adaptive learning, and the updated feature vertices are used as channel weights, thereby adjusting the channel-level response. Furthermore, it is worth noting that while single-hop neighbor aggregation typically only captures local dependencies, multi-hop neighbor aggregation can aggregate neighbor features at multiple levels, thus better understanding long-distance dependencies between ground features. This is particularly important for detecting large-scale changes in ground features in remote sensing images. Therefore, by calculating a new weight A3 and combining it with the original weight A2, a more efficient second-order neighbor aggregation is achieved, thereby improving the accuracy of change detection and global perception capabilities.

[0023] The aforementioned SFTM primarily functions to import information from graph semantic features into the backbone features, thereby achieving adaptive updates to both the backbone features and graph semantic features. Using F... b and F g These represent the core features and graph semantic features, respectively. The core process is as follows:

[0024] First, F b and F g After standardization using the BatchNorm layer, two different nonlinear projection methods are employed to obtain Q, K, and V. For F... b To collect spatial and dimensional information, depthwise separable convolution is introduced, and then the input is divided into three sets of feature matrices along the channel dimension and flattened. On the other hand, due to F... g The graph already contains highly aggregated semantic information. Continuing to use depthwise separable convolutions might introduce unnecessary noise and affect the purity of the information. Therefore, only 1×1 two-dimensional convolutions are used to project F. g This is to avoid interfering with the original information and to reduce potential noise effects.

[0025] Subsequently, cross attention (CA) is calculated on the feature matrices obtained by projecting the two matrices. The specific calculation process is as follows:

[0026]

[0027] Where Q1, K1, V1∈R HW×C Q2, K2, V2∈R hw×C d represents the number of hidden dimensions of the head.

[0028] By using cross-attention (CA), graph semantic features F can be achieved. g and main features F b Information exchange between them, thereby enabling the graph semantic features F g The information in the middle is integrated into the main feature F b In contrast to standard self-attention, cross-attention (CA) has the advantage of bidirectional information enhancement, thereby capturing global dependencies and high-level semantic information, and optimizing semantic understanding through refined feature information. Secondly, this mechanism utilizes complementary information from two input types, significantly improving model performance through synergistic effects.

[0029] The SF-A and SF-B modules used in step c are used to aggregate the features obtained from skip connections into the backbone features or graph semantic features, thereby completing the fusion of multi-scale information and improving the model's representation and generalization capabilities.

[0030] The loss function used in step d is a hybrid loss function based on the Dice loss function and the cross-entropy loss function. The specific formula is as follows:

[0031] L(pred,gt)=L dice (pred,gt)+L wce (pred,gt) (9)

[0032] Where pred represents the predicted value and gt represents the actual value.

[0033] A method and system for detecting changes in remote sensing images based on Transformer and graph semantic guidance, comprising a processor and a storage device, wherein the storage device stores multiple instructions for the processor to load and execute the steps of any of the methods described above.

[0034] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: The remote sensing image change detection method and system based on Transformer and graph semantic guidance incorporates a novel graph semantic module (GSM) in the encoding stage, aiming to guide the network to perform more detailed feature extraction through graph semantic features. In this module, each channel of the feature map is regarded as a feature vertex, and the relationship between them is represented by an adjacency matrix. Multi-hop neighbor aggregation enhances the model's ability to capture long-distance dependencies, thereby obtaining more informative graph semantic features. These graph semantic features will be used in the subsequent semantic fusion Transformer module (SFTM) to guide the backbone features extracted by the network and adaptively update both. This method makes the final change detection results of dual-temporal remote sensing images more reliable, robust, and accurate. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating the implementation process of a remote sensing image change detection method based on Transformer and graph semantic guidance according to an embodiment of the present invention.

[0036] Figure 2 This is a graph semantic module (GSM) of a remote sensing image change detection method based on Transformer and graph semantic guidance according to an embodiment of the present invention.

[0037] Figure 3 This invention relates to a semantic fusion Transformer module SFTM, which is a remote sensing image change detection method based on Transformer and graph semantic guidance.

[0038] Figure 4 This refers to the LEVER optical remote sensing image change dataset and detection results used in this embodiment of the invention. Figure 4 (a), (b), (c), (d), and (e) represent the image before the change, the image after the change, the changed reference image, the STANet detection result, the BIT detection result, and the detection result of the present invention, respectively. Detailed Implementation

[0039] A remote sensing image change detection method based on Transformer and graph semantic guidance includes: an overall framework, a graph semantic module (GSM), and a semantic fusion Transformer module (SFTM). In the overall framework, multi-layer serial-parallel connections of CNN and Transformer layers complement the information extracted by CNN and Transformer, thereby balancing network efficiency and performance. In the encoding stage, a novel graph semantic module (GSM) is constructed to guide the network in more detailed feature extraction through graph semantic features. In this module, each channel of the feature map is treated as a feature vertex, and the relationship between them is represented using an adjacency matrix. Multi-hop neighbor aggregation enhances the model's ability to capture long-distance dependencies, thus obtaining richer graph semantic features. These graph semantic features are used in the subsequent semantic fusion Transformer module (SFTM) to guide the backbone features extracted by the network and adaptively update both. SFTM mainly consists of a cross-attention (CA) layer for information exchange, a local information compensation layer to compensate for local information loss, and a reinforcement feedforward network to optimize the input. After the encoding and decoding process is completed, a 1×1 convolution is used to adjust the number of channels of the image features to two, thus obtaining the final semantic binary transformation map.

[0040] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0041] like Figure 1 The diagram shown illustrates the implementation process of the remote sensing image change detection based on Transformer and graph semantic guidance according to the present invention, including the following steps:

[0042] Step 1: Use the remote sensing software ENVI to perform image registration on the dual-time images. Through two steps of coarse correction and fine correction, the processed images T1 and T2 are obtained.

[0043] For geometric coarse correction, the relevant functions in ENVI 4.8 software are used. The specific operation steps are as follows: (1) Display the reference image and the image to be corrected; (2) Collect ground control points (GCPs); GCPs should be evenly distributed in the entire image, and the number of GCPs should be at least greater than or equal to 9. (3) Calculate the error; (4) Select the polynomial model; (5) Use bilinear interpolation to resample the output.

[0044] For the bilinear interpolation method, if we want to find the value of the unknown function f at point P = (x, y), assume that the function f is known at point Q. 11 = (x1, y1), Q 12 = (x1, y2), Q 21 = (x2, y1), and Q 22 = the values ​​of the four points (x2, y2). If we choose a coordinate system such that the coordinates of these four points are (0, 0), (0, 1), (1, 0), and (1, 1), then the bilinear interpolation formula can be expressed as:

[0045] f(x,y)≈f(0,0)(1-x)(1-y)+f(1,0)x(1-y)+f(0,1)(1-x)y+f(1,1)xy (1)

[0046] For geometric fine correction, the multispectral remote sensing image data that has undergone geometric coarse correction is subjected to geometric fine correction using automatic matching and triangulation methods.

[0047] For the triangulation method, a point-by-point insertion method is used to construct a Delaunay triangulation network. For each triangle, the row and column numbers of its three vertices and the geographic coordinates of the corresponding points in the reference image are used to determine the affine transformation model parameters inside the triangle, thereby correcting the image and obtaining the corrected remote sensing image.

[0048] Step 2: Input the two time-series images T1 and T2 into the network. Then, encode the two time-series images separately. In the first two layers of the network encoding stage, two convolutional layers are used for shallow feature extraction. Afterwards, the Graph Semantic Module (GSM) is used to generate graph semantic features, which guide the backbone features for more detailed feature extraction. The specific structure is as follows... Figure 2As shown.

[0049] Specifically, during the forward propagation, 3×3 convolutions are used to project the input features into two different subspaces. Next, global average pooling transforms the semantic projection dimension to 1×1×hw, where h and w represent the height and width of the specified graph semantic feature, and hw represents its number of channels. These channels are then used as feature vertices, and the semantic projection information is enhanced by the Graph Construction Module (GCM). Then, softmax normalization is applied to the semantic encoded map to adjust the weight of each pixel to an appropriate range. Finally, matrix multiplication is used to multiply and sum the weighted features and weight maps to generate a low-resolution graph semantic feature with the specified dimension. This graph semantic feature contains higher-level abstract information and can effectively guide the backbone features for more detailed feature extraction.

[0050] The Graph Construction Module (GCM) is the core of this method, drawing upon the theory of graph convolutional networks. In the spatial dimension of a graph convolutional network, for a vertex v... i The graph convolution operation is represented as follows:

[0051]

[0052] Where i and j represent vertex indices, X out (·) and X in (·) represent the output and input features, respectively. B i Indicates the output vertex v i The convolutional sampling set, which is defined as the set of samples with v i Adjacent vertices v j w represents the connection between each output vertex v. i and all input vertices v j The weight vector corresponding to the relationship between C(v) j This indicates that the weights of each subset are balanced.

[0053] Since the feature maps in graph convolutional networks are tensors of shape F×N (F represents the features in the vertices, and N represents the number of vertices), equation (1) can be rewritten as follows:

[0054]

[0055] Where W is the weight of the one-dimensional convolutional layer, used to represent the set of weights w for each vertex in equation (1). It is an N×N matrix, and This is a standardized operation. Here, A represents an N×N adjacency matrix, D represents an N×N degree matrix, and each element D... ii=∑ j A ij +ε. And A ij This represents the connection relationship from vertex i to vertex j, such as the existence of an adjacency relationship or the strength of the connection. ε represents a tiny value used to avoid division by zero.

[0056] Therefore, each channel of a feature can be used as a feature vertex to achieve nonlocal operations on the feature. Since the feature distribution differs across layers, the relationships between features are not easily obtained, so the adjacency matrix A cannot be manually set. Considering that the established graph structure should depend on the data, a novel graph construction method is proposed. This method can adaptively learn and model the dependencies between feature vertices, thereby effectively capturing the complex features of ground feature changes. The core idea of ​​this method can be simply expressed by the following equation:

[0057] X out =WX in (A1×A2×A3+A4) (4)

[0058] The core of this formula lies in how to model the relationships between feature vertices using an adjacency matrix. This matrix can be divided into four parts: A1, A2, A3, and A4, described in detail below:

[0059] 1) A1 represents an N×N identity matrix. Since the adjacency relationships between each feature vertex are unknown, A1 is used to represent the feature vertex itself.

[0060] 2) A2 represents an N×N diagonal matrix, which is the weight of each feature vertex calculated during the first neighborhood aggregation process. Specifically, the input tensor first passes through a 1×1 one-dimensional convolutional layer and is normalized to between 0 and 1. These values ​​are then arranged diagonally to serve as the weights for each feature vertex. The formula for calculating A2 is as follows:

[0061] A2 = T(softmax(WX) in (5)

[0062] Where W represents the weights of a 1×1 convolutional layer, and T represents arranging the results diagonally.

[0063] 3) Similar to A2, A3 also represents an N×N diagonal matrix, which is the weight of each feature vertex calculated during the second neighborhood aggregation process. The formula for calculating A3 is as follows:

[0064] A3 = T(softmax(W(X)) in (A2)))) (6)

[0065] 4) A4 is an N×N adjacency matrix. It is optimized through backpropagation to generate relationships between any two feature vertices during training. A4 is initialized with small initial values ​​to ensure the stability of initial learning, and its element values ​​are not numerically restricted. Therefore, A4 can flexibly learn global information while customizing the dependencies between various feature vertices contained in different layers.

[0066] Step 3: In the last three layers of the network encoding stage, the Semantic Fusion Transformer (SFTM) module is used for deep feature extraction. It is worth noting that when the spatial resolution of the image is reduced to one-quarter of the original image, the backbone features and graph semantic features of the dual-time image are concatenated separately for channel adjustment. The fused features are then used by two Semantic Fusion Transformer (SFTM) modules to complete the encoding part of the entire network.

[0067] like Figure 3 As shown, SFTM mainly consists of a cross-attention (CA) layer for information exchange, a local information compensation layer (LICL) layer to compensate for local information loss, and a reinforced feed-forward network (R-FFN) for input optimization. Using F... b and F g Representing the backbone features and graph semantic features respectively, the main process of SFTM is as follows:

[0068] First, F b and F g After standardization using the BatchNorm layer, two different nonlinear projection methods are employed to obtain Q, K, and V. For F... b To collect spatial and dimensional information, depthwise separable convolution is introduced, and then the input is divided into three sets of feature matrices along the channel dimension and flattened. On the other hand, due to F... g The graph already contains highly aggregated semantic information. Continuing to use depthwise separable convolutions might introduce unnecessary noise and affect the purity of the information. Therefore, only 1×1 two-dimensional convolutions are used to project F. g This is to avoid interfering with the original information and to reduce potential noise effects.

[0069] Subsequently, cross-attention (CA) calculation is performed on the feature matrices obtained by projecting the two matrices. The specific calculation process is as follows:

[0070]

[0071] Where Q1, K1, V1∈RHW×C Q2, K2, V2∈R hw×C d represents the number of hidden dimensions of the head.

[0072] By using cross-attention (CA), graph semantic features F can be achieved. g and main features F b Information exchange between them, thereby enabling the graph semantic features F g The information in the middle is integrated into the main feature F b middle

[0073] For the graph semantic branch, after the calculation in equation (8), the graph semantic feature F is updated by applying a 1×1 convolution. g For the main branches, since attention calculation is global, local information is lost during the calculation process. Therefore, the original main branch features F are introduced through skip connections. b The result is then passed through LICL to compensate for the local information loss, and then connected to the result after attention calculation as shown in Equation (7). Here, LICL is a module composed of convolutional layers, batch normalization layers and ReLU activation functions in a simple chain. Its core is to compensate for local information through 3×3 depth convolution.

[0074] Finally, the concatenated features are passed through a 1×1 convolutional layer and an R-FFN layer to obtain the updated backbone features F. g In this design, R-FFN replaces the standard feedforward network. In a standard feedforward network, it first expands the channel dimension using a linear layer, then applies an activation function to that dimension, and finally restores the output to the original channel dimension using another linear layer. However, this approach lacks a clear mechanism to control the channel relationships of features or enhance the non-linear expressive power of features. Therefore, R-FFN is used to improve this. In R-FFN, a 1×1 convolution is used to expand the channel dimension, followed by a 3×3 depthwise convolution, a SE module, and the GELU activation function, all applied sequentially to that dimension. Finally, a 1×1 pointwise convolution restores the output to the original channel dimension. R-FFN's channel dimension processing enhances the model's expressive and learning capabilities.

[0075] Step 4: In the decoding part of the network, the change information is gradually recovered through two semantic fusion Transformer modules (SFTM) and two convolutional layers. The upsampling operation is contained in the scale fusion-A (SF-A) module and the scale fusion-B (SF-B) module.

[0076] The SF-A and SF-B modules aggregate the features obtained from skip connections into the backbone features or graph semantic features, thereby achieving multi-scale information fusion and improving the model's representation and generalization capabilities. Both modules are relatively simple, and their specific structures are as follows: Figure 1 As shown on the right.

[0077] Step 5: At the end of the network, a 1×1 convolution is used to adjust the number of channels of the image features to two, thus obtaining the final semantic binary transformation map. Furthermore, throughout the training process, a hybrid loss function based on the Dice loss function and the cross-entropy loss function is used, with the specific formula as follows:

[0078] L(pred,gt)=L dice (pred,gt)+L wce (pred,gt) (9)

[0079] Where pred represents the predicted value and gt represents the actual value.

[0080] The Dice loss function is calculated as follows:

[0081]

[0082] Among them, t n t represents the true label category of pixel n. When n is a pixel of the invariant class, t n The value is 0 when n is a variable pixel. n The value is 1; y n This represents the predicted probability that pixel n belongs to the change class.

[0083] The cross-entropy loss function is calculated as follows:

[0084]

[0085] Among them, t m y represents the true label category of pixel m. m This represents the predicted probability that pixel m belongs to the change class.

[0086] The invention will be further described in detail below with specific experimental data. The experimental data used in this experiment was LEVER change detection data, with images cropped to 256×256. The experimental platform was a Win11 system, the graphics card was an NVIDIA GeForce RTX 3060, the deep learning framework used was PyTorch, and the cuDNN library and CUDA computing architecture were used for training acceleration. During training, data augmentation techniques were applied, including Gaussian blur, random scaling, random flipping, random cropping, and random color transformation. A hybrid loss function based on the Dice loss function and cross-entropy loss function, along with the AdamW optimizer, were used for model training. In the experiment, h and w were set to 8, the batch size was set to 8, the initial learning rate was set to 0.0001, the epoch was set to 200, and a linear decay operation was used to reduce the learning rate until it reached 0 in the last epoch.

[0087] To verify the effectiveness of the present invention, the change detection method of the present invention is compared with the following change detection method:

[0088] (1) The STANet detection method proposed by Hao Chen et al. [“A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection.(Remote Sensing, 2020, 12(10)]

[0089] (2) The BIT detection method (GAN) proposed by Hao Chen et al. [The detection method proposed by Hao Chen et al. in the article "Remotesensing image change detection with transformers. (IEEE Transactions on Geoscience and Remote Sensing, 2022, 60)]

[0090] (3) The method of the present invention.

[0091] Detection performance was assessed using precision, recall, and F1 score. F1 is a comprehensive evaluation index for change detection; a value closer to 1 indicates better performance of the change detection method. The detection results are shown in Table 1.

[0092] Table 1. Performance (%) of change detection results on the LEVIR dataset.

[0093]

[0094] As shown in Table 1, the detection method proposed in this invention has the highest F1 score, which is closer to 1 than the other three detection algorithms. Furthermore, the Precision and Recall scores of this invention are the highest among the compared algorithms, also closer to 1. In summary, the change detection algorithm of this invention outperforms the other three detection methods, indicating that the change detection method proposed in this invention is effective.

[0095] Figure 4 (a) and (b) are the before and after time phase images in the LEVER dataset. Figure 4 (c) is a reference diagram for change detection. Figure 4 (d), (e), and (f) represent the change detection results of the STANet algorithm, the BIT algorithm, and the present invention, respectively. Figure 4 Based on the comparison of the reference images, the detection effect of the algorithm proposed in this invention is the best in terms of visual effect.

[0096] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A remote sensing image change detection method based on Transformer and graph semantic guidance, characterized in that, include: a. Register two remote sensing images covering the same geographical area to obtain a preprocessed dual-temporal remote sensing image; b. For the registered dual-temporal remote sensing images, two convolutional layers are used in the first two layers of the network coding stage to perform shallow feature extraction. After that, the graph semantic module (GSM) is used to generate graph semantic features to guide the backbone features to perform more detailed feature extraction. c. In the last three layers of the network encoding stage, the Semantic Fusion Transformer module SFTM is used for deep feature extraction. d. In the decoding part of the network, the change information is gradually recovered through two semantic fusion Transformer modules and two convolutional layers. The upsampling operation is included in the scale fusion-A module and the scale fusion-B module. Finally, the number of channels of the image features is adjusted to two through a 1×1 convolution, thereby obtaining the semantic binary change map. e. Throughout the training process, a hybrid loss function based on Dice loss function and cross-entropy loss function was used. By updating the loss function through gradient, the optimal change detection model can be obtained after multiple rounds of training.

2. The remote sensing image change detection method based on Transformer and graph semantic guidance according to claim 1, wherein the image registration in step a includes geometric coarse correction and geometric fine correction.

3. The remote sensing image change detection method based on Transformer and graph semantic guidance according to claim 1, characterized in that, The Graph Semantic Module (GSM) in step b can guide the network to perform more detailed feature extraction by generating graph semantic features. Its core idea can be expressed by the following formula: X out =WX in (A1×A2×A3+A4) (4) The core of this formula lies in how to model the relationships between feature vertices using an adjacency matrix. The four matrices A1, A2, A3, and A4 are described as follows: 1) A1 represents an N×N identity matrix. Since the adjacency relationship between each feature vertex is unknown, A1 is used to represent the feature vertex itself. 2) A2 represents an N×N diagonal matrix, which is the weight of each feature vertex calculated during the first neighborhood aggregation process. Specifically, the input tensor first passes through a 1×1 one-dimensional convolutional layer and is normalized to between 0 and 1. These weights are then arranged diagonally to serve as the weights of each feature vertex. The formula for calculating A2 is as follows: A2=T(softmax(WX in )) (5) Where W represents the weights of the 1×1 convolutional layer, and T represents arranging the results diagonally; 3) Similar to A2, A3 also represents an N×N diagonal matrix. It is the weight of each feature vertex calculated during the second neighborhood aggregation process. The formula for calculating A3 is as follows: A3=T(softmax(W(X in (A2)))) (6) 4) A4 is an N×N adjacency matrix used to generate the relationship between any two feature vertices during training. It can be optimized through backpropagation. A4 is given a small initial value during initialization to ensure the stability of the initial learning, and its element values ​​are not numerically restricted. A4 can flexibly learn global information and customize the dependencies between various feature vertices contained in different levels.

4. The remote sensing image change detection method based on Transformer and graph semantic guidance according to claim 1, wherein the semantic fusion Transformer module SFTM in step c is capable of adaptively and interactively updating the backbone features and graph semantic features; Specifically, the Semantic Fusion Transformer module SFTM mainly consists of a cross-attention (CA) layer for information exchange, a local information compensation (LICL) layer for compensating for local information loss, and a reinforcement feedforward (R-FFN) network for optimizing the input. b and F g Representing the backbone features and graph semantic features respectively, the main process of SFTM is as follows: First, F b and F g After standardization using the BatchNorm layer, two different nonlinear projection methods are employed to obtain Q, K, and V, for F. b We introduce depthwise separable convolutions to collect spatial and dimensional information. Then, we divide the input into three feature matrices along the channel dimension and flatten them. We then use only 1×1 two-dimensional convolutions to project F. g This is to avoid interference with the original information and reduce potential noise impact; Secondly, the cross-attention CA is calculated on the feature matrices obtained by projecting the two matrices. The specific calculation process is as follows: in, Q1, K1, V1∈R HW×C Q2, K2, V2∈R hw×C d represents the number of hidden dimensions in the head. Through cross-attention (CA), graph semantic features F can be achieved. g and main features F b Information exchange between them, thereby enabling the graph semantic features F g The information in the middle is integrated into the main feature F b middle; For the graph semantic branch, after the calculation in equation (8), the graph semantic feature F is updated by applying a 1×1 convolution. g The original backbone feature F is introduced through skip connections. b And it is passed through the Local Information Compensation Layer (LICL) to supplement the local information loss, and then it is connected to the result after the attention calculation as shown in Equation (7). The Local Information Compensation Layer (LICL) is a module composed of a convolutional layer, a batch normalization layer and a ReLU activation function. Its core is to compensate for local information through a 3×3 depth convolution. Finally, the concatenated features are passed through a 1×1 convolutional layer and an R-FFN layer to obtain the updated backbone features F. g The standard feedforward network is replaced by R-FFN. In R-FFN, 1×1 convolution is used to expand the channel dimension. Then, 3×3 depthwise convolution, SE module and GELU activation function are applied sequentially on this dimension. Finally, 1×1 pointwise convolution is used to restore the output to the original channel dimension. The processing of the channel dimension by R-FFN can improve the expressive power and learning ability of the change detection model.

5. The remote sensing image change detection method based on Transformer and graph semantic guidance according to claim 1, characterized in that, The scale fusion-A module and scale fusion-B module in step d aggregate the features obtained from skip connections into the backbone features or graph semantic features, thereby completing the fusion of multi-scale information.

6. The remote sensing image change detection method based on Transformer and graph semantic guidance according to claim 1, characterized in that, In step e, a hybrid loss function based on the Dice loss function and the cross-entropy loss function is used for gradient update, and the specific formula is as follows: L(pred,gt)=L dice (pred,gt)+L wce (pred,gt) (9) Where pred represents the predicted value and gt represents the actual value.

7. A remote sensing image change detection system based on Transformer and graph semantic guidance, characterized in that, It includes a processor and a storage device, wherein the storage device stores a plurality of instructions for the processor to load and execute the steps of the method according to any one of claims 1 to 6.

Images