Remote sensing image change detection method and system based on Transform and graph semantic guidance

Abstract

The invention discloses a remote sensing image change detection method and system based on Transform and graph semantic guidance in the technical field of image processing. In an overall framework, a CNN (Convolutional Neural Network) and a Transform multilayer are connected in series and in parallel, a novel graph semantic module (GSM) is also constructed, and the purpose is to guide the network to carry out more detailed feature extraction through graph semantic features. In the module, each channel of a feature map is regarded as a feature vertex, an adjacent matrix is used for representing the relationship between the channels, and the capturing capability of the model on the long-distance dependency relationship is enhanced through multi-hop neighbor aggregation, so that the map semantic features with richer information are obtained. The graph semantic features are used for guiding trunk features extracted by the network in a subsequent Semantic Fusion Transform Module (SFTM), and the graph semantic features and the trunk features are interactively updated in a self-adaptive mode. Wherein the SFTM is mainly composed of cross attention used for completing information exchange, a local information compensation layer used for supplementing local information loss and an enhanced feedforward network used for optimizing input. According to the method, the performance of the network can be improved, and the change detection precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote sensing image change detection method and system based on Transformer and graph semantic guidance. Background Art

[0002] Change detection in remote sensing images is to quantitatively analyze and determine the characteristics and processes of surface changes from remote sensing data of different periods. Due to different imaging conditions and the complex and diverse characteristics of objects in the scene, high-resolution remote sensing images may show different spectral characteristics with the same semantic concept at different time and spatial positions, which still poses great challenges to the change detection task. In order to accurately identify the changed areas in complex scenes, the change detection model needs to identify the high-level semantic information of the changed areas in the scene and distinguish the real changes from the complex unrelated changed areas.

[0003] By using self-attention to model the semantic relationship between pixel pairs in space and time, better detection results can be achieved. In 2020, Hao Chen et al. from Beihang University proposed STANet, a new twin network based on metric and spatiotemporal attention. It divides the network into different scales and embeds self-attention in each small area to model long-distance dependencies between pixels. It uses batch balanced contrast loss function to alleviate the problem of sample imbalance and help the network extract features rich in contextual information. However, the computational efficiency of the model is very low, the computational complexity is very high, and it grows quadratically with the number of pixels.

[0004] At present, many change detection methods based on deep convolutional neural networks (CNN) cannot effectively capture the long-distance dependencies between pixels. In this context, Hao Chen et al. from the Beijing University of Aeronautics and Astronautics proposed a bitemporal image transformer (BIT) detection algorithm, which achieved good results in change detection tasks. However, since each head of the multi-head attention in the transformer is only responsible for a subset of the input token sequence, when the channel dimension of the subset is small, the dot product of the query and key cannot form an information function, which will lead to reduced network performance. At the same time, BIT uses a direct subtraction feature fusion method that not only easily destroys the feature structure, but also may cause noise interference. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a remote sensing image change detection method and system based on Transformer and graph semantics guidance, so that the final change detection result of dual-phase multispectral remote sensing images is more accurate.

[0006] To achieve the above object, the technical solution adopted by the present invention is: a remote sensing image change detection method based on Transformer and graph semantics guidance, comprising:

[0007] a. Perform image registration on remote sensing images to obtain pre-processed dual-time remote sensing images, and transfer these images to the network proposed by the present invention.

[0008] b. In the first two layers of the network coding stage, we use two convolutional layers for shallow feature extraction; in the last three layers of the network coding stage, we use the Semantic Fusion Transformer Module (SFTM) for deep feature extraction. Before deep feature extraction, we use a Graph Semantics Module (GSM) to generate graph semantic features to guide the backbone features for more detailed feature extraction. When the spatial resolution of the image is reduced to one-fourth of the original image, we connect the backbone features and graph semantic features of the dual-time image separately for channel adjustment, and then pass the fused features through two SFTM modules to complete the encoding part of the entire network.

[0009] c. In the decoding part of the network, we gradually recover the change information through two layers of semantic fusion Transformer modules and two convolutional layers. The upsampling operation is included in the ScaleFusion-A (SF-A) module and the ScaleFusion-B (SF-B) module. Finally, we adjust the number of channels of the image feature to two through a 1×1 convolution to obtain a semantic binary change map.

[0010] d. Throughout the training process, we used a hybrid loss function (Dice loss function and cross entropy loss function) to weaken the impact of unbalanced samples. By gradient updating the loss function, we can obtain the optimal change detection model after multiple rounds of training.

[0011] The image registration in step a includes rough geometric correction and fine geometric correction.

[0012] The core idea of ​​GSM in step b is mainly based on the theory of graph convolutional networks. In GSM, we use each channel of the feature as a feature vertex to implement non-local operations on the feature. Since the feature distribution of each layer is different and the relationship between features is not easy to obtain, the adjacency matrix A cannot be set manually. The core idea of ​​this method can be simply expressed by the following formula:

[0013] X out =WX in (A1×A2×A3+A4) (4)

[0014] The core of this formula is how we use the adjacency matrix to model the relationship between feature vertices. The matrix can be divided into four parts: A1, A2, A3 and A4, which are described as follows:

[0015] 1)4 represents an N×N identity matrix. Since the adjacency relationship between each feature vertex is unknown, we use A1 to represent the feature vertex itself.

[0016] 2) A2 represents an N×N diagonal matrix, which is the weight of each feature vertex calculated in the first neighborhood aggregation process. Specifically, the input tensor first passes through a 1×1 one-dimensional convolution layer and is normalized to between 0 and 1. They are arranged diagonally as the weight of each feature vertex. The specific process of A2 is as follows:

[0017] A2=T(softmax(WX in )) (5)

[0018] Among them, W represents the weight of the 1×1 convolutional layer, and T means arranging the results diagonally.

[0019] 3) A3 is similar to A2 and also represents an N×N diagonal matrix, which is the weight of each feature vertex calculated during the second neighborhood aggregation process. The specific process of A3 is as follows:

[0020] A3=T(softmax(W(X in (A2)))) (6)

[0021] 4) A4 is an N×N adjacency matrix. It is optimized by back-propagation and is used to generate the relationship between any two feature vertices during the training process. A4 is given a small initial value at the time of initialization to ensure the stability of the initial learning, and its element values ​​are not numerically restricted. Therefore, A4 can flexibly learn global information while customizing the dependencies between various feature vertices contained in different levels. These features enable A4 to provide the model with more personalized and efficient representation learning capabilities.

[0022] Since the elements in the adjacency matrix can not only indicate the existence or non-existence of the connection relationship between nodes, but also reflect the strength of the connection, and even be used to represent unknown relationships, these expressions can be optimized through adaptive learning to form new adjacency matrices. These adjacency matrices can be used to capture the spatial connection and change relationship between objects. In this framework, the feature vertices are updated by the adjacency matrix obtained by adaptive learning, and the updated feature vertices are used as channel weights to achieve the adjustment of channel-level response. In addition, it is worth mentioning that since single-hop neighbor aggregation can usually only capture local dependencies, multi-hop neighbor aggregation can aggregate neighbor features at multiple levels, thereby better understanding the long-distance dependencies between objects, which is particularly important for the detection of large-scale changes in objects in remote sensing images. Therefore, by calculating the new weight A3 and combining it with the original weight A2, we achieve more effective second-order neighbor aggregation, thereby improving the accuracy of change detection and global perception.

[0023] The main function of the SFTM is to import the information in the graph semantic features into the backbone features, thereby completing the adaptive update of the backbone features and graph semantic features. b and F g Represent the backbone features and graph semantic features respectively. Its core process is as follows:

[0024] First, F b and F g After normalization through the BatchNorm layer, we use two different nonlinear projections to obtain Q, K, and V. b , we introduce depthwise separable convolution to collect spatial and dimensional information, and then divide the input into three sets of feature matrices in the channel dimension and flatten them. On the other hand, since F g F already contains highly aggregated graph semantic information. If we continue to use depthwise separable convolution, it may introduce unnecessary noise and affect the purity of the information. Based on this, we only use 1×1 two-dimensional convolution to project F g , to avoid interference with the original information and reduce potential noise impact.

[0025] Then, we perform cross attention (CA) calculation on the feature matrix obtained by projecting the two. The specific calculation process is as follows:

[0026]

[0027] Among them, Q1, K1, V1∈R HW×C , Q2, K2, V2∈R hw×C , d represents the number of hidden dimensions of the head.

[0028] Through cross attention, we can achieve graph semantic features F g and backbone feature F b The information exchange between them can transform the graph semantic features F g The information in is integrated into the backbone feature F b Compared with standard self-attention, the advantage of cross-attention is that it can achieve bidirectional enhancement of information, thereby capturing global dependencies and high-level semantic information, and optimizing semantic understanding through refined feature information. Secondly, this mechanism utilizes the complementary information of the two input types, significantly improving model performance through synergy.

[0029] The function of the SF-A module and SF-B module used in step c is to aggregate the features obtained by skip connection into the backbone features or graph semantic features, thereby completing the fusion of multi-scale information to improve the representation and generalization capabilities of the model.

[0030] The mixed loss function (Dice loss function and cross entropy loss function) used in step d is as follows:

[0031] L(pred,gt)=L dice (pred,gt)+L wce (pred,gt) (9)

[0032] Among them, pred represents the predicted value and gt represents the true value.

[0033] A remote sensing image change detection method and system based on Transformer and graph semantics guidance, comprising a processor and a storage device, wherein the storage device stores a plurality of instructions for the processor to load and execute the steps of any one of the above methods.

[0034] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the remote sensing image change detection method and system based on Transformer and graph semantics guidance described in this method adds a novel graph semantic module (GSM) in the encoding stage, aiming to guide the network to extract more detailed features through graph semantic features. In this module, we regard each channel of the feature graph as a feature vertex, use the adjacency matrix to represent the relationship between them, and enhance the model's ability to capture long-distance dependencies through multi-hop neighbor aggregation, thereby obtaining more informative graph semantic features. These graph semantic features will be used in the subsequent semantic fusion Transformer module (SFTM) to guide the backbone features extracted by the network, and to adaptively and interactively update the two. This method makes the final change detection results of dual-phase multispectral remote sensing images more reliable and robust, and the detection accuracy is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the implementation process of a remote sensing image change detection method based on Transformer and graph semantics guidance according to an embodiment of the present invention;

[0036] Figure 2 It is a graph semantic module (GSM) of a remote sensing image change detection method based on twin Transformer and graph semantics guidance in an embodiment of the present invention;

[0037] Figure 3 It is a semantic fusion Transformer module (SFTM) of a remote sensing image change detection method based on twin Transformer and graph semantics guidance in an embodiment of the present invention;

[0038] Figure 4 is the LEVER optical remote sensing image change data set and detection results used in the embodiment of the present invention. Figure 4 (a), (b), (c), (d), and (e) respectively represent the image before the change, the image after the change, the change reference image, the STANet detection result, the BIT detection result, and the detection result of the present invention. DETAILED DESCRIPTION

[0039] A remote sensing image change detection method based on Transformer and graph semantics guidance includes: an overall framework, a graph semantic module and a semantic fusion Transformer module. In the overall framework, we complement the information extracted by CNN and Transformer by connecting multiple layers of CNN and Transformer in series and parallel, and balance the efficiency and performance of the network. In the encoding stage, we construct a novel graph semantic module (GSM) to guide the network to extract more detailed features through graph semantic features. In this module, we regard each channel of the feature graph as a feature vertex, use the adjacency matrix to represent the relationship between them, and enhance the model's ability to capture long-distance dependencies through multi-hop neighbor aggregation, so as to obtain more informative graph semantic features. These graph semantic features will be used in the subsequent semantic fusion Transformer module (SFTM) to guide the backbone features extracted by the network, and the two will be updated interactively and adaptively. Among them, SFTM is mainly composed of cross attention for completing information exchange, local information compensation layer for supplementing local information loss, and reinforced feedforward network for optimizing input. After completing the encoding and decoding process, a 1×1 convolution is performed to adjust the number of channels of the image features to two, thereby obtaining the final semantic binary change map.

[0040] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0041] like Figure 1 FIG. 1 is a schematic diagram of the implementation process of remote sensing image change detection based on Transformer and graph semantics guidance according to the present invention, which includes the following steps:

[0042] Step 1: Use the remote sensing software ENVI to perform image registration on the dual-time images, and obtain the processed images T1 and T2 through two steps of coarse correction and fine correction.

[0043] For rough geometric correction, the relevant functions in ENVI4.8 software are used for implementation. The specific steps are as follows:

[0044] (1) Display the reference image and the image to be corrected; (2) Collect ground control points (GCPs); GCPs should be evenly distributed in the entire image, and the number of GCPs should be at least 9. (3) Calculate the error; (4) Select the polynomial model; (5) Use bilinear interpolation for resampling output.

[0045] For the bilinear interpolation method, if we want to find the value of the unknown function f at point P = (x, y), assuming that we know the function f at Q 11 =(x1,y1),Q 12 =(x1, y2), Q 21 =(x2, y1), and Q 22 =(x2,y2) The values ​​of the four points. If a coordinate system is selected so that the coordinates of these four points are (0,0), (0,1), (1,0) and (1,1), then the bilinear interpolation formula can be expressed as:

[0046] f(x,y)≈f(0,0)(1-x)(1-y)+f(1,0)x(1-y)+f(0,1)(1-x)y+f(1,1)xy (1)

[0047] For geometric fine correction, it will use automatic matching and triangulation methods to perform geometric fine correction on the multispectral remote sensing image data that has undergone geometric coarse correction.

[0048] For the triangulation method, it uses the point-by-point insertion method to construct the Delaunay triangulation network for each triangle. It uses the row and column numbers of its three vertices and the geographic coordinates of the corresponding reference image's same-name points to determine the affine transformation model parameters inside the triangle, thereby correcting the correction image and obtaining the corrected remote sensing image.

[0049] Step 2: Input the dual-time images T1 and T2 into the network. Then, encode the dual-time images separately. In the first two layers of the network encoding stage, we use two convolutional layers to perform shallow feature extraction. After that, we use the graph semantic module (GSM) to generate graph semantic features to guide the backbone features for more detailed feature extraction. The specific structure is as follows: Figure 2 shown.

[0050] Specifically, during the forward propagation process, we use 3×3 convolution to project the input features into two different subspaces. Then, the dimension of the semantic projection is converted to 1×1×hw through global average pooling, where h and w represent the height and width of the graph semantic feature we specified, and hw represents its number of channels. After that, these channels are used as feature vertices to strengthen the information of the semantic projection through the Graph Construction Module (GCM). Then, the semantic encoding map is softmax normalized to adjust the weight of each pixel to an appropriate range. Finally, through matrix multiplication, the weighted features and weight maps are multiplied and summed to generate a low-resolution graph semantic feature with the specified dimension. We believe that this graph semantic feature contains higher-level abstract information, which can effectively guide the backbone feature to perform more detailed feature extraction.

[0051] Among them, GCM is the core of this method, which draws on the theory of graph convolutional networks. In the spatial dimension of the graph convolutional network, for the vertex v i The graph convolution operation is expressed as follows:

[0052]

[0053] Among them, i and j represent vertex indices, X out (·) and X in (·) represent the output and input features respectively. B i Represents the output vertex v i The convolution sampling set is defined as i Adjacent vertices v j w represents the value associated with each output vertex v i and all input vertices v j The weight vector corresponding to the relationship between j ) means balancing the weight of each subset.

[0054] Since the feature map in the graph convolutional network is a tensor of shape F×N (F represents the features in the vertex, N represents the number of vertices), formula (1) can be rewritten as follows:

[0055]

[0056] Where W is the weight of the one-dimensional convolutional layer, which is used to represent the set of weights w of each vertex in equation (1). is an N×N matrix, and It is a standardized operation. A represents an N×N adjacency matrix, and D represents an N×N degree matrix, in which each element D ii =∑ j A ij +ε. And A ij It indicates the connection relationship from vertex i to vertex j, such as the existence of adjacency relationship or the strength of connection. ε is a small value used to avoid division by zero.

[0057] Therefore, we can use each channel of the feature as a feature vertex to implement non-local operations on the feature. Since the feature distribution of each layer is different and the relationship between features is not easy to obtain, the adjacency matrix A cannot be set manually. Considering that the graph structure we established should depend on the data, we proposed a novel graph construction method that can adaptively learn and model the dependency relationship between feature vertices, thereby effectively capturing the complex characteristics of the changes in the ground objects. The core idea of ​​this method can be simply expressed by the following formula:

[0058] X out =WX in (A1×A2×A3+A4) (4)

[0059] The core of this formula is how we use the adjacency matrix to model the relationship between feature vertices. The matrix can be divided into four parts: A1, A2, A3 and A4, which are described as follows:

[0060] 1) A1 represents an N×N identity matrix. Since the adjacency relationship between each feature vertex is unknown, we use A1 to represent the feature vertex itself.

[0061] 2) A2 represents an N×N diagonal matrix, which is the weight of each feature vertex calculated in the first neighborhood aggregation process. Specifically, the input tensor first passes through a 1×1 one-dimensional convolution layer and is normalized to between 0 and 1. They are arranged diagonally as the weight of each feature vertex. The specific process of A2 is as follows:

[0062] A2=T(softmax(WX in )) (5)

[0063] Among them, W represents the weight of the 1×1 convolutional layer, and T means arranging the results diagonally.

[0064] 3) A3 is similar to A2 and also represents an N×N diagonal matrix, which is the weight of each feature vertex calculated during the second neighborhood aggregation process. The specific process of A3 is as follows:

[0065] A3=T(softmax(W(X in (A2)))) (6)

[0066] 4) A4 is an N×N adjacency matrix. It is optimized by back-propagation and is used to generate the relationship between any two feature vertices during the training process. A4 is given a small initial value at the time of initialization to ensure the stability of the initial learning, and its element values ​​are not numerically restricted. Therefore, A4 can flexibly learn global information while customizing the dependencies between various feature vertices contained in different levels. These features enable A4 to provide the model with more personalized and efficient representation learning capabilities.

[0067] Step 3: In the last three layers of the network coding stage, we use the semantic fusion transformer module (SFTM) for deep feature extraction. It is worth noting that when the spatial resolution of the image is reduced to a quarter of the original image, we connect the backbone features and graph semantic features of the dual-time image separately for channel adjustment, and then pass the fused features through two semantic fusion transformer modules to complete the coding part of the entire network.

[0068] like Figure 3 As shown in Figure 1, SFTM is mainly composed of cross attention for information exchange, local information compensation layer (LICL) for compensating local information loss, and reinforced feed-forward network (R-FFN) for optimizing input. b and F g Represent the backbone features and graph semantic features respectively. The main process of SFTM is as follows:

[0069] First, F b and F g After normalization through the BatchNorm layer, we use two different nonlinear projections to obtain Q, K, and V. b , we introduce depthwise separable convolution to collect spatial and dimensional information, and then divide the input into three sets of feature matrices in the channel dimension and flatten them. On the other hand, since F gF already contains highly aggregated graph semantic information. If we continue to use depthwise separable convolution, it may introduce unnecessary noise and affect the purity of the information. Based on this, we only use 1×1 two-dimensional convolution to project F g , to avoid interference with the original information and reduce potential noise impact.

[0070] Then, we perform cross attention (CA) calculation on the feature matrix obtained by projecting the two. The specific calculation process is as follows:

[0071]

[0072] Among them, Q1, K1, V1∈R HW×C , Q2, K2, V2∈R hw×C , d represents the number of hidden dimensions of the head.

[0073] Through cross attention, we can achieve graph semantic features F g and backbone feature F b The information exchange between them can transform the graph semantic features F g The information in is integrated into the backbone feature F b middle

[0074] For the graph semantics branch, after the calculation of equation (8), we update the graph semantic feature F by applying a 1×1 convolution g For the trunk branch, since the attention calculation is global, local information will be lost during the calculation process. Therefore, we introduce the original trunk feature F through skip connection b , and pass it through LICL to compensate for the local information loss, and then connect it with the result after the attention calculation as shown in formula (7). Among them, LICL is a module composed of a simple series of convolutional layers, batch normalization layers and ReLU activation functions. Its core is to compensate for local information through 3×3 depth convolution.

[0075] Finally, we pass the concatenated features through a 1×1 convolutional layer and R-FFN to obtain the updated backbone feature F g. Among them, we replaced the standard feedforward network with R-FFN. In the standard feedforward network, it first expands the channel dimension through a linear layer, then uses an activation function to process it in this dimension, and finally restores the output to the original channel dimension through a linear layer. However, this operation does not have a clear mechanism to control the channel relationship of features or enhance the nonlinear expression ability of features. Therefore, this paper proposes R-FFN to improve this situation. In R-FFN, we use 1×1 convolution to expand the channel dimension, and then use 3×3 deep convolution, SE module and GELU activation function to process it in this dimension, and finally restore the output to the original channel dimension through 1×1 point-by-point convolution. We believe that R-FFN's processing process in the channel dimension can better improve the model's expression and learning capabilities.

[0076] Step 4: In the decoding part of the network, we gradually recover the change information through two layers of semantic fusion Transformer modules and two convolutional layers. The upsampling operation is included in the scale fusion-A (SF-A) module and the scale fusion-B (SF-B) module.

[0077] The function of the SF-A module and the SF-B module is to aggregate the features obtained by the jump connection into the backbone features or graph semantic features, thereby completing the fusion of multi-scale information to improve the representation and generalization capabilities of the model. Both modules are relatively simple, and their specific structures are as follows: Figure 1 Shown on the right.

[0078] Step 5: At the end of the network, we adjust the number of channels of the image features to two through a 1×1 convolution to obtain the final semantic binary change map. In addition, during the entire training process, we used a hybrid loss function (Dice loss function and cross entropy loss function) to weaken the impact of unbalanced samples. The specific formula is as follows:

[0079] L(pred,gt)=L dice (pred,gt)+L wce (pred,gt) (9)

[0080] Among them, pred represents the predicted value and gt represents the true value.

[0081] The Dice loss function is calculated as follows:

[0082]

[0083] Among them, t n represents the true label category of pixel n. When n is a pixel of the same class, t n The value is 0. When n is a change pixel, t nThe value is 1; n represents the predicted probability that pixel n belongs to the change class.

[0084] The cross entropy loss function is calculated as follows:

[0085]

[0086] Among them, t m represents the true label category of pixel m, y m represents the predicted probability that pixel m belongs to the change class.

[0087] The present invention is further described in detail below in conjunction with specific experimental data. The experimental data used in this experiment is LEVER change detection data, and the image size is cropped to 256× 256. The experimental platform is Win11 system, the graphics card is NVIDIAGeForce RTX 3060, the deep learning framework used is PyTorch framework, and the cuDNN library and CUDA computing architecture are used for training acceleration. During the training process, data enhancement techniques were applied, including Gaussian blur, random scaling, random flipping, random cropping, and random color transformation, and a mixed loss function (Dice loss function and cross entropy loss function) and AdamW optimizer were used for model training. In the experiment, h and w were set to 8, the batch size was set to 8, the initial learning rate was set to 0.0001, the epoch was set to 200, and a linear decay operation was used to reduce the learning rate until it dropped to 0 in the last period.

[0088] In order to verify the effectiveness of the present invention, the change detection method of the present invention is compared with the following change detection methods:

[0089] (1) STANet detection method proposed by Hao Chen et al. [“A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection.” (Remote Sensing, 2020, 12(10)]

[0090] (2) BIT detection method (GAN) proposed by Hao Chen et al. [Hao Chen et al. Detection method proposed in the article "Remotesensing image change detection with transformers. (IEEE Transactions on Geoscience and Remote Sensing, 2022, 60]

[0091] (3) The method of the present invention.

[0092] The detection performance is measured using precision, recall, and F1 score indicators. F1 is a comprehensive evaluation indicator for change detection. The closer its value is to 1, the better the performance of the change detection method. The detection results are shown in Table 1.

[0093] Table 1. Change detection results performance on LEVIR dataset (%)

[0094]

[0095] As can be seen from Table 1, the F1 score of the detection method proposed in the present invention is the largest, which is closer to 1 than the other three detection algorithms. In addition, the Precision and Recall of the present invention are the largest among the comparison algorithms and are closer to 1. In summary, the performance of the change detection algorithm of the present invention is better than the other three detection methods, which shows that the change detection method proposed in the present invention is effective.

[0096] Figure 4 (a) and (b) are the before and after phase images in the LEVER dataset. Figure 4 (c) is the reference image for change detection. Figure 4 (d), (e), and (f) are the change detection results of the STANet algorithm, the BIT algorithm, and the present invention, respectively. Figure 4 Compared with the reference image, the detection effect of the algorithm proposed in the present invention is the best in terms of visual effect.

[0097] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A remote sensing image change detection method based on Transformer and graph semantics guidance, characterized in that: include: a. Performing image registration on remote sensing images to obtain pre-processed dual-time remote sensing images, and transferring these images into the network proposed by the present invention; b. For the registered dual-phase optical remote sensing images, we use two convolutional layers to perform shallow feature extraction in the first two layers of the network coding stage. After that, we use the graph semantic module (GSM) to generate graph semantic features to guide the backbone features for more detailed feature extraction; c. In the last three layers of the network coding stage, we use the semantic fusion transformer module (SFTM) for deep feature extraction; d. In the decoding part of the network, we gradually recover the change information through two layers of semantic fusion Transformer modules and two convolutional layers. The upsampling operation is included in the scale fusion-A (SF-A) module and the scale fusion-B (SF-B) module. Finally, we adjust the number of channels of the image feature to two through a 1×1 convolution to obtain a semantic binary change map. e. Throughout the training process, we used a hybrid loss function (Dice loss function and cross entropy loss function) to weaken the impact of unbalanced samples. By updating the loss function with gradients, we can obtain the optimal change detection model after multiple rounds of training.

2. According to claim 1, the image registration in step a includes rough geometric correction and fine geometric correction.

3. According to claim 1, the GSM in step b can guide the network to extract more detailed features by generating graph semantic features. The core idea can be simply expressed by the following formula: X out =WX in (A1×A2×A3+A4) (4) The core of this formula is how we use the adjacency matrix to model the relationship between feature vertices. The matrix can be divided into four parts: A1, A2, A3 and A4, which are described as follows: 1) A1 represents an N×N identity matrix. Since the adjacency relationship between each feature vertex is unknown, we use A1 to represent the feature vertex itself. 2) A2 represents an N×N diagonal matrix, which is the weight of each feature vertex calculated in the first neighborhood aggregation process. Specifically, the input tensor first passes through a 1×1 one-dimensional convolution layer and is normalized to between 0 and 1. They are arranged diagonally as the weight of each feature vertex. The specific process of A2 is as follows: A2=T(softmax(WX in )) (5) Among them, W represents the weight of the 1×1 convolutional layer, and T means arranging the results diagonally. 3) A3 is similar to A2 and also represents an N×N diagonal matrix, which is the weight of each feature vertex calculated during the second neighborhood aggregation process. The specific process of A3 is as follows: <h2 style=";text-align:left;direction:ltr">A3 = T(softmax(W(X<h2 style=";text-align:left;direction:ltr"> in <h2 style=";text-align:left;direction:ltr"> (A2))) (6) 4) A4 is an N×N adjacency matrix. It is optimized by back-propagation and is used to generate the relationship between any two feature vertices during the training process. A4 is given a small initial value at the time of initialization to ensure the stability of the initial learning, and its element values ​​are not numerically restricted. Therefore, A4 can flexibly learn global information while customizing the dependencies between various feature vertices contained in different levels. These features enable A4 to provide the model with more personalized and efficient representation learning capabilities.

4. According to claim 1, the SFTM in step c is capable of interactively and adaptively updating the backbone features and graph semantic features. Specifically, SFTM is mainly composed of a cross attention (CA) for completing information exchange, a local information compensation layer (LICL) for compensating local information loss, and a reinforced feedforward network (R-FFN) for optimizing the input. b and F g Represent the backbone features and graph semantic features respectively. The main process of SFTM is as follows: First, F b and F g After normalization through the BatchNorm layer, we use two different nonlinear projections to obtain Q, K, and V. b , we introduce depthwise separable convolution to collect spatial and dimensional information, and then divide the input into three sets of feature matrices in the channel dimension and flatten them. On the other hand, since F g F already contains highly aggregated graph semantic information. If we continue to use depthwise separable convolution, it may introduce unnecessary noise and affect the purity of the information. Based on this, we only use 1×1 two-dimensional convolution to project F g , to avoid interference with the original information and reduce potential noise impact. Then, we perform cross attention (CA) calculation on the feature matrix obtained by projecting the two. The specific calculation process is as follows: in, Q1,K1,V1∈R HW×C , Q2, K2, V2∈R hw×C , d represents the number of hidden dimensions of the head. Through cross attention, we can achieve graph semantic features F g and backbone feature F b The information exchange between them can transform the graph semantic features F g The information in is integrated into the backbone feature F b middle For the graph semantics branch, after the calculation of equation (8), we update the graph semantic feature F by applying a 1×1 convolution g For the trunk branch, since the attention calculation is global, local information will be lost during the calculation process. Therefore, we introduce the original trunk feature F through skip connection b , and pass it through LICL to compensate for the local information loss, and then connect it with the result after the attention calculation as shown in formula (7). Among them, LICL is a module composed of a simple series of convolutional layers, batch normalization layers and ReLU activation functions. Its core is to compensate for local information through 3×3 depth convolution. Finally, we pass the concatenated features through a 1×1 convolutional layer and R-FFN to obtain the updated backbone feature F g Among them, we use R-FFN instead of the standard feedforward network. In R-FFN, we use 1×1 convolution to expand the channel dimension, and then use 3×3 depth convolution, SE module and GELU activation function to process it in this dimension, and finally restore the output to the original channel dimension through 1×1 point-by-point convolution. We believe that R-FFN's processing process in the channel dimension can better improve the model's expressiveness and learning ability.

5. According to claim 1, the SF-A module and the SF-B module in step d are used to aggregate the features obtained by the jump connection into the backbone features or graph semantic features, thereby completing the fusion of multi-scale information to improve the representation and generalization capabilities of the model.

6. According to claim 1, the mixed loss function (Dice loss function and cross entropy loss function) in step e is used for gradient update. The specific formula is as follows: L(pred,gt)=L dice (pred,gt)+L wce (pred,gt) (9) in, Pred represents the predicted value, and gt represents the true value.

7. A remote sensing image change detection system based on Transformer and graph semantics guidance, characterized in that: It comprises a processor and a storage device, wherein the storage device stores a plurality of instructions for the processor to load and execute the steps of the method according to any one of claims 1 to 7.

Images