Remote sensing image change detection method and device based on graph convolution and computer equipment

Abstract

The application relates to a remote sensing image change detection method and device based on graph convolution and a computer device in the technical field of image processing. The method is used for labeling obtained double-time-phase remote sensing images to obtain training samples, constructing a remote sensing image change detection network, and training the training samples; the trained remote sensing image change detection network is used for detecting double-time-phase remote sensing images to be measured to obtain a remote sensing image change detection result. The network comprises two feature extraction branches composed of backbone networks and EF modules which are the same in structure and parameters, a graph convolution coding module, two graph convolution decoding modules which are the same in structure, and an output network; the feature extraction branches are used for extracting image features and boundary information of the training samples, the graph convolution coding module is used for coding features after double-time-phase image fusion, and the graph convolution decoding modules are used for fusing multi-level feature differences and predicting change masks. The method can accurately predict the change area of a picture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a remote sensing image change detection method and device based on graph convolution and computer equipment. BACKGROUND

[0002] Change detection is an important task in remote sensing image analysis, which assigns a binary label (i.e., changed or unchanged) to each pixel in the region by comparing two temporal images of the same region taken at different times. Remote sensing image change detection has achieved remarkable success based on the powerful discrimination ability of deep neural networks, but it is still difficult to well aggregate context information for high-resolution remote sensing change detection tasks. SUMMARY

[0003] Therefore, it is necessary to provide a remote sensing image change detection method and device based on graph convolution and computer equipment to solve the above technical problems.

[0004] A remote sensing image change detection method based on graph convolution, the method comprises:

[0005] Obtaining a double-time remote sensing image, and labeling the double-time remote sensing image to obtain a training sample.

[0006] Constructing a remote sensing image change detection network based on graph convolution; the remote sensing image change detection network comprises two feature extraction branches composed of a backbone network and a boundary perception fusion module, both of which have the same structure and parameters, a graph convolution encoding module, two graph convolution decoding modules with the same structure, and an output network; the feature extraction branch is used to extract the features of the training sample through the backbone network, splice the obtained image features to obtain fused features, extract boundary information at multiple scales through the shallow backbone network by using the boundary perception fusion module, and splice the image features and the boundary information to obtain single-time image features; the graph convolution encoding module is used to encode the fused features by using graph convolution to obtain double-time image encoding features; the graph convolution decoding module is used to decode the single-time image features and the double-time image encoding features by using graph convolution to obtain image decoding features; and the output network is used to perform up-sampling operation on the difference feature map of the image decoding features output by the two graph convolution decoding modules to obtain a change detection prediction result.

[0007] Training the remote sensing image change detection network by using the label of the training sample and the change detection prediction result obtained by inputting the training sample into the remote sensing image change detection network, to obtain a trained remote sensing image change detection network.

[0008] The trained remote sensing image change detection network is used for detecting the to-be-detected double-time-phase remote sensing images to obtain a remote sensing image change detection result.

[0009] In one of the embodiments, the training samples include first-time-phase remote sensing image training samples and second-time-phase remote sensing image training samples.

[0010] The remote sensing image change detection network is trained by using the labels of the training samples and the change detection prediction results obtained by inputting the training samples into the remote sensing image change detection network to obtain the trained remote sensing image change detection network, including:

[0011] The first-time-phase remote sensing image training samples and the second-time-phase remote sensing image training samples are respectively input into two feature extraction branches to obtain fusion features, first-time-phase remote sensing image features and second-time-phase remote sensing image features.

[0012] The fusion features are input into the graph convolution encoding module to obtain double-time-phase image encoding features.

[0013] The first-time-phase remote sensing image features and the double-time-phase image encoding features are input into a first graph convolution decoding module to obtain first-time-phase image decoding features.

[0014] The second-time-phase remote sensing image features and the double-time-phase image encoding features are input into a second graph convolution decoding module to obtain second-time-phase image decoding features.

[0015] The first-time-phase image decoding features and the second-time-phase image decoding features are input into an output network to obtain change detection prediction results.

[0016] The remote sensing image change detection network is reversely trained according to the change detection prediction results and the labels of the training samples to obtain the trained remote sensing image change detection network.

[0017] In one of the embodiments, the backbone network in the feature extraction branch is obtained by removing a classification head of a Resnet50 network.

[0018] The first-time-phase remote sensing image training samples and the second-time-phase remote sensing image training samples are respectively input into two feature extraction branches to obtain fusion features, first-time-phase remote sensing image features and second-time-phase remote sensing image features, including:

[0019] The first-time-phase remote sensing image training samples are input into a backbone network of the first feature extraction branch to obtain first-time-phase remote sensing image features.

[0020] Input the features of the first to third layers of the backbone network into a boundary perception fusion module of the first feature extraction branch to obtain first-phase remote sensing image boundary information.

[0021] Splice the first-phase remote sensing image features and the first-phase remote sensing image boundary information to obtain first-phase image features.

[0022] Input the second-phase remote sensing image training sample into the second feature extraction branch to obtain second-phase remote sensing image features and second-phase image features.

[0023] Splice the first-phase remote sensing image features and the second-phase remote sensing image features to obtain fusion features.

[0024] In one of the embodiments, input the features of the first to third layers of the backbone network into a boundary perception fusion module of the first feature extraction branch to obtain first-phase remote sensing image boundary information, comprising:

[0025] Input the features of the first to third layers of the backbone network into a boundary perception fusion module of the first feature extraction branch, and perform channel fusion on the features of the second layer of the backbone network after up-sampling and the features of the first layer of the backbone network to obtain first intermediate features.

[0026] Perform channel fusion on the features of the third layer of the backbone network after up-sampling and the features of the second layer of the backbone network to obtain second intermediate features.

[0027] Perform channel fusion on the second intermediate features after up-sampling and the first intermediate features, and perform point convolution processing on the fusion result to obtain first-phase remote sensing image boundary information.

[0028] In one of the embodiments, the graph convolution encoding module comprises a spatial attention graph convolution encoding branch and a channel attention graph convolution branch; wherein the spatial attention graph convolution encoding branch comprises a step convolution layer, a spatial attention graph convolution module, a nearest neighbor interpolation module and a point convolution layer; and the channel attention graph convolution branch comprises two linear transformation functions, a channel attention graph convolution module and a point convolution layer.

[0029] In the graph convolution encoding module:

[0030] After processing the fusion features through the step convolution layer, new features X s are obtained.

[0031] Input the new features X s into the spatial attention graph convolution module to obtain spatial graph convolution features M s .

[0032]

[0033] wherein, μ(·) is a linear transformation, φ(·) is a linear transformation, represents an adjacency matrix, W s is a parameter matrix of the spatial attention graph convolution.

[0034] The spatial graph convolution feature M s is multiplied by the fusion feature after being processed by the nearest neighbor interpolation method, and then is processed by a point convolution layer to obtain the output of the spatial attention graph convolution encoding branch.

[0035] The fusion feature is processed by a first linear transformation and a second linear transformation respectively, and the first linear transformation result and the second linear transformation result are multiplied to obtain a new feature X f .

[0036] The new feature X f is input into the channel convolution attention graph convolution module to obtain a channel graph convolution feature M f is:

[0037] M f = (I + A f )X f W f

[0038] wherein, I is an identity matrix, I + A f is an adjacency matrix, W f is a parameter matrix of the channel attention graph convolution.

[0039] The second linear transformation result is multiplied by the channel graph convolution feature M f and then is input into a point convolution layer to obtain the output of the channel attention graph convolution branch.

[0040] The output of the spatial attention graph convolution encoding branch and the output of the channel attention graph convolution branch are spliced to obtain a dual-phase image encoding feature.

[0041] In one of the embodiments, the graph convolution decoding module comprises a spatial attention graph convolution decoding branch and a channel attention graph convolution decoding branch; wherein the spatial attention graph convolution decoding branch comprises two step convolution layers, a spatial attention graph convolution decoding module and a point convolution layer; and the channel attention graph convolution decoding branch comprises two linear transformations, a spatial attention graph convolution decoding module and a point convolution layer.

[0042] In the first graph convolution decoding module:

[0043] The first phase image feature and the double-phase image coding feature are respectively input into a step convolution layer to obtain new image features Y s and new coding features X s .

[0044] The new image features Y s and the new coding features X s are input into a spatial attention graph convolution decoding module to obtain spatial graph convolution features M s are as follows:

[0045]

[0046] Wherein, μ(·), φ(·) are linear transformations, represents an adjacency matrix, and W s is a parameter matrix of the spatial attention graph convolution decoding module.

[0047] The spatial graph convolution features M s and the new image features Y s are multiplied and then passed through a point convolution layer to obtain spatial attention graph convolution decoding branch outputs.

[0048] The first phase image feature and the double-phase image coding feature are respectively transformed linearly, and the obtained transformation results are multiplied to obtain features X_Y T .

[0049] The features X_Y T are input into the spatial attention graph convolution decoding module to obtain channel graph convolution features M f are as follows:

[0050] M f ′=(I+A f ′)(X_Y T )W f ′

[0051] Wherein, I is an identity matrix, I+A f ′ is an adjacency matrix, and W f ′ is a parameter matrix of the channel attention graph convolution decoding module.

[0052] The channel graph convolution features M f and the new image features Y s are multiplied and then passed through a point convolution layer to obtain channel attention graph convolution decoding branch outputs.

[0053] The spatial attention graph convolution decoding branch outputs and the channel attention graph convolution decoding branch outputs are spliced to obtain first phase image decoding features.

[0054] In one of the embodiments, the linear transformation is implemented by a 1x1 convolution layer.

[0055] In one of the embodiments, the first time-phase image decoding feature and the second time-phase image decoding feature are input into an output network to obtain a change detection prediction result, including:

[0056] The first time-phase image decoding feature and the second time-phase image decoding feature are input into an output network to obtain a change detection prediction result, including:

[0057] Pre=σ(upsample(|P 1 -P 2 |))

[0058] wherein, Pre is the change detection prediction result, P 1 ,P 2 respectively represent the first time-phase image decoding feature and the second time-phase image decoding feature, sigma is an activation function, and upsample is an up-sampling operation.

[0059] A remote sensing image change detection device based on graph convolution, comprising:

[0060] A training sample acquisition module is configured to acquire double time-phase remote sensing images, label the double time-phase remote sensing images, and obtain training samples.

[0061] A remote sensing image change detection network based on graph convolution is configured to construct a remote sensing image change detection network based on graph convolution; the remote sensing image change detection network comprises two feature extraction branches composed of a backbone network and a boundary perception fusion module, a graph convolution encoding module, two graph convolution decoding modules with the same structure, and an output network; the feature extraction branch is configured to extract features of the training samples through the backbone network, splice the obtained image features to obtain fused features, extract boundary information at multiple scales through the shallow backbone network by using the boundary perception fusion module, splice the image features and the boundary information to obtain single time-phase image features; the graph convolution encoding module is configured to encode the fused features by using graph convolution to obtain double time-phase image encoding features; the graph convolution decoding module is configured to decode the single time-phase image features and the double time-phase image encoding features by using graph convolution to obtain image decoding features; and the output network is configured to perform up-sampling operation on the difference feature map of the image decoding features output by the two graph convolution decoding modules to obtain a change detection prediction result.

[0062] The remote sensing image change detection network training module is configured to train the remote sensing image change detection network by using the labels of the training samples and change detection prediction results obtained by inputting the training samples into the remote sensing image change detection network, and obtain a trained remote sensing image change detection network.

[0063] The remote sensing image change detection module is configured to detect the to-be-detected dual-time-phase remote sensing images by using the trained remote sensing image change detection network, and obtain a remote sensing image change detection result.

[0064] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0065] The remote sensing image change detection method, device and computer device based on the graph convolution, the method includes: acquiring dual-time-phase remote sensing images, labeling the dual-time-phase remote sensing images, obtaining training samples, and constructing a remote sensing image change detection network, the network includes two feature extraction branches composed of residual networks and boundary perception fusion modules which are the same in structure and parameters, a graph convolution encoding module, two graph convolution decoding modules which are the same in structure, and an output network; the feature extraction branch is configured to extract image features of the training samples by using a backbone network, extract boundary information at multiple scales by using the boundary perception fusion module to the shallow layer features of the backbone network, the graph convolution encoding module is configured to further refine the feature information after the fusion of the dual-time-phase images, and the graph convolution decoding module is configured to fuse the multi-level feature differences and predict a change mask. The remote sensing image change detection network is trained by using the training samples, and the trained remote sensing image change detection network is used to detect the to-be-detected dual-time-phase remote sensing images, and obtain a remote sensing image change detection result. The method can accurately predict the change area of the image. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 The figure is a flowchart of the remote sensing image change detection method based on the graph convolution in one embodiment;

[0067] Figure 2 The figure is a structural diagram of the remote sensing image change detection network based on the graph convolution in another embodiment;

[0068] Figure 3 The figure is a structural diagram of the boundary perception fusion module in another embodiment;

[0069] Figure 4 The figure is a structural diagram of the graph convolution decoding module in another embodiment;

[0070] Figure 5 The figure is a structural block diagram of the remote sensing image change detection device based on the graph convolution in one embodiment;

[0071] Figure 6 Figure 1 is a schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0072] For the purposes of the present application, the technical solutions and advantages thereof are more clearly apparent, the following will be further described in detail in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0073] The remote sensing image change detection network based on graph convolution is referred to as BI-GCN.

[0074] The edge perception fusion module is referred to as EF module.

[0075] In one embodiment, as shown in Figure 1 A remote sensing image change detection method based on graph convolution is provided, and the method comprises the following steps:

[0076] Step 100: Acquire double-time remote sensing images, and label the double-time remote sensing images to obtain training samples.

[0077] The double-time remote sensing images are two remote sensing images taken at different times of the same place.

[0078] Step 102: Construct a remote sensing image change detection network based on graph convolution.

[0079] The remote sensing image change detection network comprises two feature extraction branches composed of a backbone network and an edge perception fusion module, a graph convolution encoding module, two graph convolution decoding modules with the same structure, and an output network.

[0080] The feature extraction branch is used to extract the features of the training samples through the backbone network, splice the obtained image features to obtain fusion features, use the edge perception fusion module to extract boundary information at multiple scales through the shallow backbone network, and splice the image features and the boundary information to obtain single-time image features.

[0081] The graph convolution encoding module is used to encode the fusion features using graph convolution to obtain double-time image encoding features.

[0082] The graph convolution decoding module is used to decode the single-time image features and the double-time image encoding features using graph convolution to obtain image decoding features.

[0083] The output network is used to perform an upsampling operation on the difference feature map of the image decoding features output by the two graph convolution decoding modules to obtain a change detection prediction result.

[0084] Specifically, the structure of the remote sensing image change detection network based on graph convolution is as shown in the following figure. Figure 2 The backbone network is the feature extraction part of the Resnet network, and the boundary perception fusion module (EF module) extracts boundary information at multiple scales through the shallow backbone network. The graph convolution encoding module further refines the feature information after the fusion of the dual-time images, and the graph convolution decoding module is used to fuse the multi-level feature differences and predict the change mask.

[0085] In standard convolution, information only communicates between multiple positions in a small neighborhood defined by the filter size (e.g., 3*3 size convolution). In order to create a large receptive field and capture long-range dependencies, it is necessary to stack multiple layers one after another as in common CNN architectures. Graph convolution is an efficient and easy-to-integrate module that encapsulates the neighborhood definition used in standard convolution and allows long-range information exchange in a single network layer by defining edges between nodes in a graph. Formally, graph convolution is defined as

[0086] X = σ (AXW) (1)

[0087] where σ(·) is a nonlinear activation function, A is an adjacency matrix representing the neighborhood relationship between nodes of the graph, W is a parameter matrix, and the definition and structure of the graph play a key role in determining information propagation. The graph convolution encoding module constructs an orthogonal graph space through different graph projection strategies to better learn the difference features in the change detection task.

[0088] The overall architecture of the graph convolution decoding module follows the design idea of the graph convolution encoding module.

[0089] Step 104: training the remote sensing image change detection network using the labels of the training samples and the change detection prediction results obtained by inputting the training samples into the remote sensing image change detection network, to obtain a trained remote sensing image change detection network.

[0090] Step 106: detecting the to-be-detected dual-time remote sensing images using the trained remote sensing image change detection network to obtain a remote sensing image change detection result.

[0091] The remote sensing image change detection method based on the graph convolution comprises the following steps: acquiring double-time-phase remote sensing images, labeling the double-time-phase remote sensing images to obtain training samples, and constructing a remote sensing image change detection network, wherein the network comprises two feature extraction branches composed of residual networks and boundary perception fusion modules which are the same in structure and parameters, a graph convolution encoding module, two graph convolution decoding modules which are the same in structure, and an output network; the feature extraction branch is used to extract image features of the training samples by using a backbone network, boundary information of multiple scales is extracted from the shallow layer features of the backbone network through the boundary perception fusion module, the feature information of the double-time-phase images after fusion is further refined by the graph convolution encoding module, and the graph convolution decoding module is used to fuse the multi-level feature differences and predict a change mask. The remote sensing image change detection network is trained by using the training samples, and the trained remote sensing image change detection network is used to detect the double-time-phase remote sensing images to be measured to obtain a remote sensing image change detection result. The method can accurately predict the change area of the images.

[0092] In one embodiment, the training samples comprise first-time-phase remote sensing image training samples and second-time-phase remote sensing image training samples; and step 104 comprises the following specific steps:

[0093] Step 200: inputting the first-time-phase remote sensing image training samples and the second-time-phase remote sensing image training samples into two feature extraction branches respectively to obtain fusion features, first-time-phase remote sensing image features, and second-time-phase remote sensing image features.

[0094] Step 202: inputting the fusion features into the graph convolution encoding module to obtain double-time-phase image encoding features.

[0095] Step 204: inputting the first-time-phase image features and the double-time-phase image encoding features into the first graph convolution decoding module to obtain first-time-phase image decoding features.

[0096] Step 206: inputting the second-time-phase remote sensing image features and the double-time-phase image encoding features into the second graph convolution decoding module to obtain second-time-phase image decoding features.

[0097] Step 208: inputting the first-time-phase image decoding features and the second-time-phase image decoding features into the output network to obtain a change detection prediction result.

[0098] Step 210: reversely training the remote sensing image change detection network according to the change detection prediction result and the label of the training samples to obtain a trained remote sensing image change detection network.

[0099] In one of the embodiments, the backbone network in the feature extraction branch is obtained by removing the classification head of the Resnet50 network; step 200 comprises: inputting the first-phase remote sensing image training sample into the backbone network of the first feature extraction branch to obtain first-phase remote sensing image features; inputting the features of the first to third layers of the backbone network into the boundary perception fusion module of the first feature extraction branch to obtain first-phase remote sensing image boundary information; splicing the first-phase remote sensing image features and the first-phase remote sensing image boundary information to obtain first-phase image features; inputting the second-phase remote sensing image training sample into the second feature extraction branch to obtain second-phase remote sensing image features and second-phase image features; splicing the first-phase remote sensing image features and the second-phase remote sensing image features to obtain fusion features.

[0100] In one of the embodiments, inputting the features of the first to third layers of the backbone network into the boundary perception fusion module of the first feature extraction branch to obtain first-phase remote sensing image boundary information comprises: inputting the features of the first to third layers of the backbone network into the boundary perception fusion module of the first feature extraction branch, performing channel fusion on the features of the second layer of the backbone network after up-sampling and the features of the first layer of the backbone network to obtain first intermediate features; performing channel fusion on the features of the third layer of the backbone network after up-sampling and the features of the second layer of the backbone network to obtain second intermediate features; performing channel fusion on the second intermediate features after up-sampling and the first intermediate features, and performing point convolution processing on the fusion result to obtain first-phase remote sensing image boundary information. The structural diagram of the boundary perception fusion module is shown in Figure 3 .

[0101] Specifically, most network models need to down-sample the input image to meet the memory and speed requirements, but some boundary information will be lost in the down-sampling process, and there is a problem that the extracted features are not sufficient, resulting in incomplete regions and irregular boundaries. This situation occurs particularly in shallow networks, resulting in poor performance. The boundary information of the shallow layer should not be ignored in the change detection field, and the method proposes a boundary perception fusion module based on the extraction of boundary information by the shallow network and the auxiliary graph convolution decoding module to fuse the features to enhance the ability of the model to learn features from the shallow network. The formula is as follows:

[0102] Y e = τ(Y 128 , up(τ(Y 64 , up(Y 32 ))) (2)

[0103] wherein Y 128 , Y 64 , Y 32are the first, second and third layer features of resnet50 respectively, τ(·) represents a channel fusion operation, and up(·) represents an up-sampling operation by a factor of two.

[0104] Y e The channel fused features extracted by the channel pruning and backbone network are input into a graph convolution decoding module.

[0105] In one embodiment, as shown in FIG. 1, the graph convolution encoding module comprises a spatial attention graph convolution encoding branch and a channel attention graph convolution branch. Figure 2 The spatial attention graph convolution encoding branch comprises a stride convolution layer, a spatial attention graph convolution module, a nearest neighbor interpolation module and a point convolution layer. The channel attention graph convolution branch comprises two linear transformation functions, a channel attention graph convolution module and a point convolution layer. In the graph convolution encoding module, the fused features are processed by the stride convolution layer to obtain new features X s . The new features X s are input into the spatial attention graph convolution module to obtain spatial graph convolution features M s .

[0106]

[0107] wherein μ(·) and φ(·) are linear transformations, A represents an adjacency matrix, and W s is a parameter matrix of the spatial attention graph convolution.

[0108] The spatial graph convolution features M s are processed by the nearest neighbor interpolation method and then multiplied by the fused features, and then processed by the point convolution layer to obtain the output of the spatial attention graph convolution encoding branch. The fused features are processed by the first linear transformation and the second linear transformation, respectively, and the first linear transformation result and the second linear transformation result are multiplied to obtain new features X f . The new features X f are input into the channel attention graph convolution module to obtain channel graph convolution features M f .

[0109] M f = (I + A f )X f W f (4) wherein I is an identity matrix, I + A f is an adjacency matrix, and W f is a parameter matrix of the channel attention graph convolution. The adjacency matrix A f and W f are randomly initialized and optimized and learned by the gradient descent algorithm in the end-to-end network training process.​

[0110] The second linear transformation result is multiplied with the channel graph convolution feature M f and input into a point convolution layer to obtain an output of the channel attention graph convolution branch; and the output of the spatial attention graph convolution encoding branch and the output of the channel attention graph convolution branch are spliced to obtain a dual-phase image encoding feature.

[0111] Specifically, the upper branch of the graph convolution encoding module is the spatial attention graph convolution encoding branch, in which: the input feature is projected into a new coordinate space Ω S , and a down-sampling operation is used to convert the input feature X into a new feature X s , with a down-sampling rate of 8. In the coordinate space Ω S , three learnable linear transformations μ(·), φ(·) are used to generate the components of graph convolution. The specific formula is shown in formula (3).

[0112] Then, the M s is projected back to the original coordinate space based on the input feature X through a nearest neighbor interpolation algorithm interp(·).

[0113]

[0114] The upper branch of the graph convolution encoding module is the channel attention graph convolution branch, in which: the feature space graph convolution models the correlation of channels along the channel dimension of the network, and captures the correlation between more abstract features in the image. The input feature X is reduced by 8 times and 4 times the feature channel number of the input feature X through linear transformations α(·), β(·), respectively. The input feature X is projected into a feature space f through a projection function X f =α(X)β(X) T to obtain a new feature X f . According to formula 1, the expression of the new feature X f is shown in formula (4). Then, the new feature M f is mapped back to the original feature space The calculation result of the final refined feature is:

[0115]

[0116] Wherein, X' is a dual-phase image encoding feature, τ represents a channel fusion operation, and conv represents a point convolution.

[0117] In one embodiment, as Figure 4 ​As shown, the graph convolution decoding module comprises a spatial attention graph convolution decoding branch and a channel attention graph convolution decoding branch; wherein the spatial attention graph convolution decoding branch comprises two step convolution layers, a spatial attention graph convolution decoding module and a point convolution layer; the channel attention graph convolution decoding branch comprises two linear transformations, a spatial attention graph convolution decoding module and a point convolution layer; in the first graph convolution decoding module: the first time phase image feature and the dual time phase image coding feature are respectively input into the step convolution layer to obtain new image features Y s ′ and new coding features X s ′; the new image features Y s ′ and the new coding features X s ′ are input into the spatial attention graph convolution decoding module to obtain spatial graph convolution features M s ′, which are as follows:

[0118]

[0119] wherein μ(·), φ(·) are linear transformations, represents an adjacency matrix, and W s ′ is a parameter matrix of the spatial attention graph convolution decoding module.

[0120] After the spatial graph convolution features M s ′ and the new image features Y s ′ are multiplied and then pass through the point convolution layer, the output of the spatial attention graph convolution decoding branch is obtained; the first time phase image feature and the dual time phase image coding feature are respectively transformed through linear transformation, and the obtained transformation results are multiplied to obtain a feature X_Y T ; the feature X_Y T is input into the spatial attention graph convolution decoding module to obtain channel graph convolution features M f ′, which are as follows:

[0121] M f ′=(I+A f ′)(X_Y T )W f ′(8) wherein I is an identity matrix, I+A f ′ is an adjacency matrix, and W f ′ is a parameter matrix of the channel attention graph convolution decoding module.

[0122] After the channel graph convolution features M f ′ and the new image features Y s ′ are multiplied and then pass through the point convolution layer, the output of the channel attention graph convolution decoding branch is obtained; the output of the spatial attention graph convolution decoding branch and the output of the channel attention graph convolution decoding branch are spliced to obtain the first time phase image decoding feature.

[0123] Specifically, the overall architecture of the graph convolution decoding module follows the design idea of the graph convolution encoding module. Taking the first graph convolution decoding module as an example to analyze the calculation formula of feature extraction, the features input into the first graph convolution decoding module include: the first time phase image feature Y1 and the dual time phase image encoding feature.

[0124] The formula of the coordinate space attention part is as follows, taking the fusion information of as the adjacency matrix, M s ′ is projected back to the original coordinate space with the input feature Y1 as the reference.

[0125]

[0126] Wherein, Y1 is the first time phase image feature.

[0127] The two linear transformations a(·), b(·) in the channel attention graph convolution decoding branch respectively reduce the feature channel number of the input feature X' by 8 times and the feature Y i by 2 times, and then map the new feature M f ′ back to the original feature space with the input feature Y1 as the reference.

[0128] X_Y T =a(X')b(Y1) T (11)

[0129] M f ′=(I+A f )(X_Y T )W f (12)

[0130]

[0131] The final decoding calculation result is:

[0132]

[0133] The structure and principle of the second graph convolution decoding module are the same as those of the first graph convolution decoding module. The second time phase image feature Y2 and the dual time phase image encoding feature are input into the second graph convolution decoding module to obtain the second time phase image decoding feature P 2 ; P 1 and P 2 represent the prediction results of a pair of pictures in the dual time phase.

[0134] In one embodiment, the linear transformation is realized by a 1x1 convolution layer.

[0135] In one embodiment, step 208 includes: inputting the first temporal image decoding features and the second temporal image decoding features into the output network to obtain the change detection prediction result:

[0136] Pre=σ(upsample(|P 1 -P 2 |)) (15)

[0137] Where Pre is the change detection prediction result, and P 1 ,P 2 These represent the decoding features of the first and second phase images, respectively. σ is the activation function, preferably a ReLU function, and upsample is the upsampling operation.

[0138] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0139] In a validation embodiment, the experimental data utilized the LEVIR-CD open dataset for model training and testing. The LEVIR-CD dataset contains 637 pairs of 1024×1024 high-resolution (0.5m) Google Earth images from two time periods, with changed building labels. To reduce computational cost, the original dataset images were uniformly cropped to 256×256 pixels. The experimental environment configuration is shown in Table 1.

[0140] Table 1 Experimental Environment Configuration

[0141] Experimental environment Configuration description Server GPU NVDIAA10 Memory 20GB Learning framework torch 1.10.0, cuda11.2 Development language Python 3.9.12 Editor Pycharm

[0142] To demonstrate the recognition performance of the BI-GCN model on the dataset, comparative experiments were conducted using the SiamUnet_diff network model, the SiamUnet_conc network model, and the BIT network model. The models were evaluated using five metrics: precision, recall, F1 score, intersection-over-union ratio (IoU), and overall accuracy (OA).

[0143] Table 2 Comparison of various indicators

[0144]

[0145]

[0146] In one embodiment, as shown in Figure 5 a graph convolution-based remote sensing image change detection device is provided, comprising: a training sample acquisition module, a graph convolution-based remote sensing image change detection network construction module, a graph convolution-based remote sensing image change detection network training module, and a remote sensing image change detection module, wherein:

[0147] The training sample acquisition module is configured to acquire double-time-phase remote sensing images, and label the double-time-phase remote sensing images to obtain training samples.

[0148] The graph convolution-based remote sensing image change detection network construction module is configured to construct a graph convolution-based remote sensing image change detection network; the remote sensing image change detection network comprises two feature extraction branches composed of a backbone network and a boundary perception fusion module, a graph convolution encoding module, two graph convolution decoding modules with the same structure, and an output network; the feature extraction branch is configured to extract features of the training samples through the backbone network, splice the obtained image features to obtain fusion features, extract boundary information at multiple scales through the shallow backbone network by using the boundary perception fusion module, and splice the image features and the boundary information to obtain single-time-phase image features; the graph convolution encoding module is configured to encode the fusion features by using graph convolution to obtain double-time-phase image encoding features; the graph convolution decoding module is configured to decode the single-time-phase image features and the double-time-phase image encoding features by using graph convolution to obtain image decoding features; and the output network is configured to perform up-sampling operation on the difference feature maps of the image decoding features output by the two graph convolution decoding modules to obtain a change detection prediction result.

[0149] The graph convolution-based remote sensing image change detection network training module is configured to train the remote sensing image change detection network by using the labels of the training samples and the change detection prediction results obtained by inputting the training samples into the remote sensing image change detection network, to obtain a trained remote sensing image change detection network.

[0150] The remote sensing image change detection module is configured to detect the double-time-phase remote sensing images to be measured by using the trained remote sensing image change detection network to obtain a remote sensing image change detection result.

[0151] In one of the embodiments, the training sample includes a first time phase remote sensing image training sample and a second time phase remote sensing image training sample; the remote sensing image change detection network based on graph convolution training module is further configured to input the first time phase remote sensing image training sample and the second time phase remote sensing image training sample into two feature extraction branches respectively to obtain the fusion feature, the first time phase remote sensing image feature and the second time phase remote sensing image feature; input the fusion feature into the graph convolution coding module to obtain the double time phase image coding feature; input the first time phase image feature and the double time phase image coding feature into the first graph convolution decoding module to obtain the first time phase image decoding feature; input the second time phase remote sensing image feature and the double time phase image coding feature into the second graph convolution decoding module to obtain the second time phase image decoding feature; input the first time phase image decoding feature and the second time phase image decoding feature into the output network to obtain the change detection prediction result; perform reverse training on the remote sensing image change detection network according to the change detection prediction result and the label of the training sample to obtain the trained remote sensing image change detection network.

[0152] In one of the embodiments, the backbone network in the feature extraction branch is obtained by removing the classification head of the Resnet50 network; the remote sensing image change detection network based on graph convolution training module is further configured to input the first time phase remote sensing image training sample into the backbone network of the first feature extraction branch to obtain the first time phase remote sensing image feature; input the features of the first to third layers of the backbone network into the boundary perception fusion module of the first feature extraction branch to obtain the first time phase remote sensing image boundary information; splice the first time phase remote sensing image feature and the first time phase remote sensing image boundary information to obtain the first time phase image feature; input the second time phase remote sensing image training sample into the second feature extraction branch to obtain the second time phase remote sensing image feature and the second time phase image feature; splice the first time phase remote sensing image feature and the second time phase remote sensing image feature to obtain the fusion feature.

[0153] In one of the embodiments, the remote sensing image change detection network based on graph convolution training module is further configured to input the features of the first to third layers of the backbone network into the boundary perception fusion module of the first feature extraction branch, perform channel fusion on the features of the second layer of the backbone network after up-sampling and the features of the first layer of the backbone network to obtain the first intermediate feature; perform channel fusion on the features of the third layer of the backbone network after up-sampling and the features of the second layer of the backbone network to obtain the second intermediate feature; perform channel fusion on the second intermediate feature after up-sampling and the first intermediate feature, and perform point convolution processing on the fusion result to obtain the first time phase remote sensing image boundary information.

[0154] In one embodiment, the graph convolutional coding module includes a spatial attention graph convolutional coding branch and a channel attention graph convolutional branch; wherein, the spatial attention graph convolutional coding branch includes a stride convolutional layer, a spatial attention graph convolutional module, a nearest neighbor interpolation module, and a point convolutional layer; the channel attention graph convolutional branch includes two linear transformation functions, a channel attention graph convolutional module, and a point convolutional layer; the graph convolution-based remote sensing image change detection network training module is further used to: process the fused features through the stride convolutional layer to obtain a new feature X in the graph convolutional coding module. s ; to add new feature X s The input is fed into the spatial attention map convolution module to obtain the spatial map convolution feature M. s Spatial graph convolutional features M s The expression for is shown in equation (3).

[0155] Convolution features M from spatial graph s After processing using nearest neighbor interpolation, the result is multiplied by the fused feature, and then processed by a point convolutional layer to obtain the output of the spatial attention map convolutional coding branch. The fused feature is then processed by the first and second linear transformations respectively, and the results of the first and second linear transformations are multiplied to obtain a new feature X. f ; to add new feature X f The input is fed into the channel convolution attention map convolution module to obtain the channel map convolution features M. f Channel map convolutional features M f The expression for is shown in equation (4).

[0156] The result of the second linear transformation is combined with the channel map convolution feature M. f After multiplication, the output is fed into a point convolutional layer to obtain the output of the channel attention map convolutional branch. The outputs of the spatial attention map convolutional coding branch and the channel attention map convolutional branch are then concatenated to obtain the dual-temporal image coding features.

[0157] In one embodiment, the graph convolutional decoding module includes a spatial attention graph convolutional decoding branch and a channel attention graph convolutional decoding branch; wherein, the spatial attention graph convolutional decoding branch includes two stride convolutional layers, a spatial attention graph convolutional decoding module, and a point convolutional layer; the channel attention graph convolutional decoding branch includes two linear transformations, a spatial attention graph convolutional decoding module, and a point convolutional layer; the graph convolution-based remote sensing image change detection network training module is further configured to, in the first graph convolutional decoding module, input the first temporal image features and the dual-temporal image encoding features into the stride convolutional layer respectively to obtain new image features Y. s ′ and new coding feature X s ′;Transfer new image features Y s ′ and new coding feature Xs Input into the spatial attention graph convolution decoding module to obtain spatial graph convolution feature M s ′, the expression of the spatial graph convolution feature M s ′ is shown in equation (7).

[0158] After multiplying the spatial graph convolution feature M s ′ and the new image feature Y s ′ and passing through the point convolution layer, the spatial attention graph convolution decoding branch output is obtained; the first time phase image feature and the dual time phase image coding feature are respectively transformed by linear transformation, and the obtained transformation results are multiplied to obtain the feature X_Y T ; the feature X_Y T is input into the spatial attention graph convolution decoding module to obtain the channel graph convolution feature M f ′, the expression of the channel graph convolution feature M f ′ is shown in equation (8).

[0159] After multiplying the channel graph convolution feature M f ′ and the new image feature Y s ′ and passing through the point convolution layer, the channel attention graph convolution decoding branch output is obtained; the spatial attention graph convolution decoding branch output and the channel attention graph convolution decoding branch output are spliced to obtain the first time phase image decoding feature.

[0160] In one of the embodiments, the linear transformation is realized by a 1x1 convolution layer.

[0161] In one of the embodiments, the graph convolution based remote sensing image change detection network training module is further configured to input the first time phase image decoding feature and the second time phase image decoding feature into the output network to obtain the change detection prediction result, and the expression of the change detection prediction result is shown in equation (15).

[0162] The specific limitations of the graph convolution based remote sensing image change detection device can be referred to the limitations of the graph convolution based remote sensing image change detection method in the above, which will not be repeated here. Each module in the above graph convolution based remote sensing image change detection device can be realized by software, hardware and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to each module by the processor.

[0163] In one embodiment, a computer device is provided, which can be a terminal, and the internal structure diagram thereof can be as shown in Figure 6As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a graph convolution-based remote sensing image change detection method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0164] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0165] In one embodiment, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0166] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0167] The above-described embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A method for detecting changes in remote sensing images based on graph convolution, characterized in that, The method includes: Acquire dual-temporal remote sensing images and annotate them to obtain training samples; A remote sensing image change detection network based on graph convolution is constructed. This network includes two feature extraction branches consisting of a backbone network and a boundary-aware fusion module with identical structure and parameters, a graph convolutional encoding module, two graph convolutional decoding modules with identical structures, and an output network. The feature extraction branches extract features from training samples through the backbone network, concatenate the obtained image features to obtain fused features, and use the boundary-aware fusion module to extract boundary information at multiple scales through a shallow backbone network, concatenating the image features with the boundary information to obtain single-temporal image features. The graph convolutional encoding module encodes the fused features using graph convolution to obtain dual-temporal image encoded features. The graph convolutional decoding module decodes the single-temporal image features and the dual-temporal image encoded features using graph convolution to obtain image decoded features. The output network performs upsampling based on the difference feature maps of the image decoded features output by the two graph convolutional decoding modules to obtain change detection prediction results. The remote sensing image change detection network is trained using the annotation of training samples and the change detection prediction results obtained by inputting the training samples into the remote sensing image change detection network, resulting in a trained remote sensing image change detection network. A trained remote sensing image change detection network is used to detect changes in the two-phase remote sensing images under test, and the results of remote sensing image change detection are obtained.

2. The method according to claim 1, characterized in that, The training samples include training samples of remote sensing images from the first time phase and training samples of remote sensing images from the second time phase. The remote sensing image change detection network is trained using the labeled training samples and the change detection prediction results obtained by inputting the training samples into the network. The resulting trained network includes: The first and second time-phase remote sensing image training samples are respectively input into two feature extraction branches to obtain fused features, first time-phase remote sensing image features and second time-phase remote sensing image features; The fused features are input into the graph convolutional coding module to obtain dual-temporal image coding features; The first temporal remote sensing image features and the dual temporal image coding features are input into the first graph convolutional decoding module to obtain the first temporal image decoding features; The second temporal remote sensing image features and the dual-temporal image coding features are input into the second graph convolutional decoding module to obtain the second temporal image decoding features; The first temporal image decoding features and the second temporal image decoding features are input into the output network to obtain the change detection prediction results; The remote sensing image change detection network is back-trained based on the change detection prediction results and the annotations of the training samples to obtain a trained remote sensing image change detection network.

3. The method according to claim 2, characterized in that, The backbone network in the feature extraction branch is obtained by removing the classification head from the ResNet50 network. The training samples of the first and second temporal remote sensing images are respectively input into two feature extraction branches to obtain fused features, first temporal remote sensing image features, and second temporal remote sensing image features, including: The training samples of the first temporal remote sensing image are input into the backbone network of the first feature extraction branch to obtain the features of the first temporal remote sensing image. The features of the first to third layers of the backbone network are input into the boundary-aware fusion module of the first feature extraction branch to obtain the boundary information of the first temporal remote sensing image; The first phase remote sensing image features are stitched together with the boundary information of the first phase remote sensing image to obtain the first phase image features; The training samples of the second temporal remote sensing image are input into the second feature extraction branch to obtain the features of the second temporal remote sensing image and the features of the second temporal image. The features of the first and second temporal remote sensing images are stitched together to obtain fused features.

4. The method according to claim 3, characterized in that, The features of the first to third layers of the backbone network are input into the boundary-aware fusion module of the first feature extraction branch to obtain the boundary information of the first temporal remote sensing image, including: The features of the first to third layers of the backbone network are input into the boundary-aware fusion module of the first feature extraction branch. The features of the second layer of the backbone network are upsampled and then fused with the features of the first layer of the backbone network to obtain the first intermediate feature. The features of the third layer of the backbone network are upsampled and then fused with the features of the second layer of the backbone network to obtain the second intermediate feature. After upsampling the second intermediate feature, it is fused with the first intermediate feature through channels. The fusion result is then processed by point convolution to obtain the boundary information of the first temporal remote sensing image.

5. The method according to claim 2, characterized in that, The graph convolutional coding module includes: a spatial attention graph convolutional coding branch and a channel attention graph convolutional branch; wherein, the spatial attention graph convolutional coding branch includes: a stride convolutional layer, a spatial attention graph convolutional module, a nearest neighbor interpolation module, and a point convolutional layer; the channel attention graph convolutional branch includes two linear transformation functions, a channel attention graph convolutional module, and a point convolutional layer; In the graph convolutional coding module: The fused features are processed through a stride convolutional layer to obtain a new feature X. s ; New feature X s The input is fed into the spatial attention map convolution module to obtain the spatial map convolution feature M. s : Among them, μ(·), φ(·) are all linear transformations. W represents the adjacency matrix. s Let be the parameter matrix for the spatial attention map convolution; The spatial graph convolution feature M s After processing with the nearest neighbor interpolation method, the result is multiplied with the fused feature, and then processed by a point convolutional layer to obtain the output of the spatial attention map convolutional coding branch. The fused features are processed using the first and second linear transformations respectively. The results of the first and second linear transformations are then multiplied to obtain the new feature X. f ; New feature X f The input is fed into the channel attention map convolution module to obtain the channel map convolution feature M. f for: M f =(I+A f )X f W f Where I is the identity matrix, I+A f Let W be the adjacency matrix. f The parameter matrix for the channel attention map convolution; The result of the second linear transformation is combined with the channel map convolution feature M. f After multiplication, the result is fed into a point convolutional layer to obtain the output of the channel attention map convolutional branch; By concatenating the outputs of the spatial attention map convolutional coding branch and the channel attention map convolutional branch, a dual-temporal image coding feature is obtained.

6. The method according to claim 3, characterized in that, The graph convolution decoding module includes: a spatial attention graph convolution decoding branch and a channel attention graph convolution decoding branch; wherein, the spatial attention graph convolution decoding branch includes two stride convolutional layers, a spatial attention graph convolution decoding module, and a point convolutional layer; the channel attention graph convolution decoding branch includes two linear transformations, a spatial attention graph convolution decoding module, and a point convolutional layer; In the first graph convolution decoding module: The first temporal image features and the dual-temporal image coding features are respectively input into the stride convolutional layer to obtain new image features Y. s ′ and new coding feature X s ′; New image features Y s ′ and new coding feature X s The input is fed into the spatial attention map convolutional decoding module to obtain the spatial map convolutional features M. s 'for: Among them, μ(·), φ(·) are all linear transformations. W represents the adjacency matrix. s ′ represents the parameter matrix of the spatial attention map convolutional decoding module; The spatial graph convolution feature M s ′ and the new image feature Y s After multiplication, the result is passed through a point convolutional layer to obtain the output of the spatial attention map convolutional decoding branch; The first temporal image features and the dual-temporal image encoded features are respectively subjected to linear transformations, and the resulting transformations are multiplied to obtain features X_Y. T ; The feature X_Y T The input is fed into the spatial attention map convolutional decoding module to obtain the channel map convolutional features M. f 'for: M f ′=(I+A f ′)(X_Y T )W f ′ Where I is the identity matrix, I+A f ′ is the adjacency matrix, W f ' represents the parameter matrix of the channel attention map convolutional decoding module; Convolution feature M of the channel map f ′ and the new image feature Y s After multiplication, the result is passed through a point convolutional layer to obtain the output of the channel attention map convolutional decoding branch. The outputs of the spatial attention map convolutional decoding branch and the channel attention map convolutional decoding branch are concatenated to obtain the first temporal image decoding features.

7. The method according to claim 5 or 6, characterized in that, The linear transformation is achieved through a 1×1 convolutional layer.

8. The method according to claim 2, characterized in that, The first temporal image decoding features and the second temporal image decoding features are input into the output network to obtain change detection prediction results, including: The first temporal image decoding features and the second temporal image decoding features are input into the output network to obtain the change detection prediction result: Pre=σ(upsample(|P 1 -P 2 |)) Where Pre is the change detection prediction result, and P 1 ,P 2 σ represents the decoding features of the first phase image and the decoding features of the second phase image, respectively. σ is the activation function and upsample is the upsampling operation.

9. A remote sensing image change detection device based on graph convolution, characterized in that, The device includes: The training sample acquisition module is used to acquire dual-temporal remote sensing images and annotate the dual-temporal remote sensing images to obtain training samples; A module for constructing a graph convolution-based remote sensing image change detection network is provided. This network comprises two feature extraction branches consisting of a backbone network and a boundary-aware fusion module with identical structure and parameters, a graph convolutional encoding module, two graph convolutional decoding modules with identical structures, and an output network. The feature extraction branches extract features from training samples through the backbone network, concatenate the resulting image features to obtain fused features, and use the boundary-aware fusion module to extract boundary information at multiple scales through a shallow backbone network, concatenating the image features with the boundary information to obtain single-temporal image features. The graph convolutional encoding module encodes the fused features using graph convolution to obtain dual-temporal image encoded features. The graph convolutional decoding module decodes the single-temporal image features and the dual-temporal image encoded features using graph convolution to obtain image decoded features. The output network performs upsampling based on the difference feature maps of the image decoded features output by the two graph convolutional decoding modules to obtain change detection prediction results. The training module for the remote sensing image change detection network based on graph convolution is used to train the remote sensing image change detection network using the annotation of training samples and the change detection prediction results obtained by inputting the training samples into the remote sensing image change detection network, so as to obtain a trained remote sensing image change detection network. The remote sensing image change detection module is used to detect changes in the dual-temporal remote sensing images under test using a trained remote sensing image change detection network, and obtain the remote sensing image change detection results.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

Images