Unsupervised cross-sensor change detection method and device based on graph structure learning

Abstract

The application relates to an unsupervised multi-source change detection method and device based on graph structure learning. The method comprises the following steps: constraining the training process of a graph autoencoder according to a pre-set reconstruction loss function, a change alignment loss function and a sparsity loss function, so as to obtain a trained graph autoencoder; performing graph structure learning on the initial adjacency matrices of pre-event images and post-event images by using the trained graph autoencoder, so as to obtain an optimized backward structural difference vector in a pre-event field and a forward structural difference vector in a post-event field; generating a forward difference image and a backward difference image according to the optimized backward structural difference vector in the pre-event field and the forward structural difference vector in the post-event field; fusing and segmenting the forward difference image and the backward difference image, so as to obtain a binary change image; and the binary change image is a multi-source change detection result. The method can improve the change detection accuracy of multi-source remote sensing images.

Description

Technical Field

[0001] This application relates to the field of heterogeneous change detection technology, and in particular to an unsupervised heterogeneous change detection method and apparatus based on graph structure learning. Background Technology

[0002] Heterogeneous change detection (HCD) is a current focus in remote sensing, attracting widespread attention. Heterogeneous remote sensing images originate from different sensors and cannot be directly compared to detect changes. The goal of remote sensing image change detection is to identify areas of change by jointly analyzing two (or more) remote sensing images of the same scene acquired at different times. Change detection plays a crucial role in Earth observation applications such as disaster assessment, urban development, and environmental monitoring. Heterogeneous change detection is a common problem in remote sensing because it is impossible to detect changes by directly comparing remote sensing images from different domains.

[0003] However, in recent years, nonlocal similarity within heterogeneous remote sensing images has been used to detect changes between these images. Current nonlocal similarity-based methods rely on constructing a nonlocal K-nearest neighbor (KNN) graph and then using graph mapping to compute structural differences between forward and backward directions. However, this approach suffers from two main problems. First, the KNN graph may fail to accurately capture the topological relationships between real land cover, especially in high-resolution and complex scenes involving multiple land cover types and different scales. Second, since forward and backward structural differences are computed in independent domains, variations inherently exist between them. In most current methods, constructing the nonlocal KNN graph, computing structural differences, and generating the change map is a unidirectional process, limited to fusing forward and backward structural differences at the back end. However, relying solely on back-end fusion leads to poor-quality change maps when the quality of structural differences in one direction is poor. Furthermore, relying solely on back-end fusion cannot fundamentally affect the graph construction process and fails to fully utilize the complementary information in the forward and backward structural differences, resulting in low accuracy in heterogeneous remote sensing image change detection. Summary of the Invention

[0004] Therefore, it is necessary to provide an unsupervised heterogeneous change detection method based on graph structure learning that can improve the accuracy of image change detection, in order to address the above-mentioned technical problems.

[0005] An unsupervised heterogeneous change detection method based on graph structure learning, the method comprising:

[0006] Obtain the image before and after the event to be detected;

[0007] The pre-event and post-event images are co-segmented using a simple linear iterative clustering algorithm, resulting in identical superpixel segmentation results for both images.

[0008] The mean and variance of the superpixels are extracted from the superpixel segmentation results and used as the features of the superpixels to construct an initial feature matrix;

[0009] The initial adjacency matrix of the pre-event image and the post-event image is obtained by calculating the superpixel segmentation results of the pre-event image and the post-event image using the KNN algorithm;

[0010] The graph autoencoder performs graph structure learning on the initial adjacency matrix and initial feature matrix to obtain the feature matrix of the hidden layer and the new adjacency matrix. The structural consistency principle is used to perform consistency calculation on the feature matrix of the hidden layer and the new adjacency matrix of the pre-event image and the post-event image to obtain the backward structural difference vector in the pre-event domain and the forward structural difference vector in the post-event domain.

[0011] The graph autoencoder is trained according to the pre-set reconstruction loss function, change alignment loss function and sparse loss function to obtain a trained graph autoencoder;

[0012] The trained graph autoencoder is used to learn the graph structure of the initial adjacency matrix of the pre-event image and the post-event image, so as to obtain the optimized backward structure difference vector in the pre-event domain and the forward structure difference vector in the post-event domain.

[0013] Generate a forward difference image and a backward difference image based on the optimized backward structure difference vector in the forward domain and the forward structure difference vector in the backward domain.

[0014] The forward and backward difference images are fused to obtain the final difference image;

[0015] Otsu thresholding is performed on the difference image to obtain a binary change image; the binary change image is the result of heterogeneous change detection.

[0016] In one embodiment, the pre-event image and the post-event image are co-segmented using a simple linear iterative clustering algorithm to obtain the same superpixel segmentation results for the pre-event image and the post-event image, including:

[0017] The pre-event and post-event images are co-segmented using a simple linear iterative clustering algorithm, resulting in identical superpixel segmentation results for both pre-event and post-event images.

[0018]

[0019]

[0020] Where, N SLet be the number of superpixels, and i and j represent different superpixel indices. The i-th superpixel in the image before the event and the image after the event are respectively denoted as . and m represents the ordinate of a pixel, n represents the x-coordinate of a pixel, M represents the height of the image, N represents the width of the image, and C represents the height of the image. X C represents the number of channels in the image before the event. Y This indicates the number of channels in the image after the event.

[0021] In one embodiment, the KNN algorithm is used to calculate the initial adjacency matrix of the pre-event and post-event images based on the superpixel segmentation results, including:

[0022] The initial adjacency matrix of the image before the event is obtained by calculating the superpixel segmentation results of the image before the event using the KNN algorithm.

[0023]

[0024]

[0025] in, Represents and Superpixel X i The set of K most similar superpixels, where the value of K is empirically chosen as... N S This represents the number of superpixels.

[0026] In one embodiment, graph structure learning is performed on the initial adjacency matrix and the initial feature matrix according to the graph autoencoder to obtain the feature matrix of the hidden layer and a new adjacency matrix, including:

[0027] The initial adjacency matrix and initial feature matrix of the image before the event are encoded using graph convolutional layers, resulting in the feature matrix of the hidden layer.

[0028]

[0029] Where σ is the ReLU nonlinear activation function, A x I is the initial adjacency matrix of the image before the event, and I is the identity matrix. yes The degree matrix, This is the weight matrix of the first graph convolutional layer. The initial feature matrix of the image before the event. This is the weight matrix of the second graph convolutional layer;

[0030] The feature matrix is ​​decoded using a decoder to obtain a new adjacency matrix.

[0031]

[0032] Where T represents the transpose operation.

[0033] In one embodiment, consistency calculations are performed on the feature matrices and new adjacency matrices of the hidden layers of the pre-event and post-event images according to the structural consistency principle, to obtain the backward structural difference vector in the pre-event domain and the forward structural difference vector in the post-event domain, including:

[0034] Based on the structural consistency principle, consistency calculations are performed on the feature matrix of the hidden layer of the pre-event image, the new adjacency matrix of the pre-event image, and the post-event image to obtain the backward structural difference vector in the pre-event domain.

[0035]

[0036] Among them, Γ x It is the distance metric matrix, Γ x Each element in the matrix represents the Euclidean distance between different feature vectors in the feature matrix of the hidden layer of the image before the event. This represents the new adjacency matrix of the image before the event. Represents the new adjacency matrix of the image after the event;

[0037] Based on the structural consistency principle, consistency calculations are performed on the feature matrix of the post-event image, the optimized adjacency matrix of the pre-event image, and the post-event image to obtain the forward structural difference vector f in the post-event domain. y .

[0038] In one embodiment, the pre-set reconstruction loss function includes a pre-set reconstruction loss function for the image before the event and a pre-set reconstruction loss function for the image after the event; the pre-set reconstruction loss function for the image before the event is...

[0039]

[0040] Where, N S A represents the number of superpixels, where i and j represent the different superpixel indices. x This represents the initial adjacency matrix.

[0041] In one embodiment, the change alignment loss function is:

[0042]

[0043] Among them, f i x f represents the backward structural difference vector corresponding to the i-th superpixel. i y The forward structure difference vector corresponding to the i-th superpixel

[0044] In one embodiment, the sparse loss function is:

[0045]

[0046] In one embodiment, the forward difference image and the backward difference image are fused to obtain a final difference image, including:

[0047] The forward and backward difference images are fused to obtain the final difference image.

[0048] DI final =(DI) x / max(DI x )+DI y / max(DI y )) / 2

[0049] Among them, DI x Represents the backward difference image, DI y This represents a forward difference image.

[0050] In one embodiment, Otsu thresholding is performed on the difference image to obtain a binary transformation image, including:

[0051] Otsu thresholding is performed on the difference image to obtain a binary transformation image.

[0052]

[0053] Where CM is a binary transformation image, and T is the threshold obtained by the Otsu method.

[0054] An unsupervised heterogeneous change detection device based on graph structure learning, the device comprising:

[0055] The superpixel segmentation module is used to acquire the pre-event image and the post-event image to be detected; it performs co-segmentation of the pre-event image and the post-event image according to a simple linear iterative clustering algorithm to obtain the same superpixel segmentation results for the pre-event image and the post-event image;

[0056] The neighbor graph calculation module is used to extract the mean and variance of superpixels from the superpixel segmentation results as the features of the superpixels to construct the initial feature matrix; the KNN algorithm is used to calculate the initial adjacency matrix of the superpixel segmentation results of the pre-event image and the post-event image.

[0057] The graph structure learning module is used to learn the graph structure of the initial adjacency matrix and initial feature matrix based on the graph autoencoder, so as to obtain the feature matrix of the hidden layer and the new adjacency matrix. The consistency principle is used to calculate the consistency of the feature matrix and the new adjacency matrix of the hidden layer of the image before the event and the image after the event, so as to obtain the backward structure difference vector in the domain before the event and the forward structure difference vector in the domain after the event.

[0058] The graph autoencoder training module is used to train the graph autoencoder according to the pre-set reconstruction loss function, change alignment loss function and sparse loss function to obtain the trained graph autoencoder.

[0059] The difference vector calculation module is used to learn the graph structure of the initial adjacency matrix of the pre-event image and the post-event image using the trained graph autoencoder, and obtain the optimized backward structure difference vector in the pre-event domain and the forward structure difference vector in the post-event domain.

[0060] The difference detection module generates forward difference images and backward difference images based on the optimized backward structure difference vector in the forward domain and the forward structure difference vector in the backward domain. The forward difference images and backward difference images are fused to obtain the final difference image. The difference image is then segmented using Otsu thresholding to obtain a binary change image. The binary change image represents the heterogeneous change detection result.

[0061] The aforementioned unsupervised heterogeneous change detection method and apparatus based on graph structure learning first performs co-segmentation of the pre-event and post-event images using a simple linear iterative clustering algorithm, obtaining identical superpixel segmentation results for both images. Then, the mean and variance of the superpixels are extracted as their features to obtain an initial feature matrix. By using superpixels as the basic unit for heterogeneous change detection, instead of individual pixels, the algorithm complexity is reduced. Furthermore, the capture of local textures and global structures in the image is improved by utilizing contextual information. The KNN algorithm is then used to construct an initial adjacency matrix to capture the structural information of the pre- and post-event images. Finally, the powerful graph representation capability of a graph autoencoder is used to optimize the initial adjacency matrix. This method leverages the ability of graph autoencoders to learn complex structures and relationships in graph data, overcoming the limitations of K-nearest neighbor graphs in capturing topological relationships between real land cover. This improves the accuracy of capturing image features. Furthermore, by setting reconstruction loss functions, change alignment loss functions, and sparsity loss functions, it incorporates prior knowledge that forward and backward structural differences should capture the same change events into the graph structure learning process. The training process of the graph autoencoder, guided by the loss functions, promotes the fusion of forward and backward structural differences and the optimization of the graph structure. This allows for more effective fusion of complementary information between forward and backward structural differences, improving the accuracy of the final difference image and thus enhancing the accuracy of the final change detection. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating an unsupervised heterogeneous change detection method based on graph structure learning in one embodiment.

[0063] Figure 2 This is a structural block diagram of an unsupervised heterogeneous change detection device based on graph structure learning in one embodiment. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] In one embodiment, such as Figure 1 As shown, an unsupervised heterogeneous change method based on graph structure learning is provided, including the following steps:

[0066] Step 102: Obtain the pre-event image and post-event image to be detected; perform co-segmentation on the pre-event image and post-event image using a simple linear iterative clustering algorithm to obtain the same superpixel segmentation results for the pre-event image and post-event image.

[0067] Identifying changes on Earth is generally done by comparing images of the same area taken at different times to identify differences. Pre-event images are images of the area taken before a particular event occurs, while post-event images are images of the area taken after the event occurs.

[0068] Assume a heterogeneous image X = x(m,n,c) | 1 ≤ m ≤ M, 1 ≤ n ≤ N, 1 ≤ c ≤ C X and Y=y(m,n,c)|1≤m≤M,1≤n≤N,1≤c≤C Y Registration was performed at the same location, where M and N represent the height and width of the image, and C... X and C Y Let X and Y represent the number of channels, respectively. Using superpixels as the basic unit for heterogeneous change detection, instead of individual pixels, reduces algorithm complexity and improves the capture of local textures and global structures in the image by utilizing contextual information. The specific segmentation method used in this application stacks X and Y together and then applies a simple linear iterative clustering algorithm for co-segmentation. This method ensures that the superpixel segmentation results in X and Y are identical. The final superpixel segmentation result is defined as:

[0069]

[0070]

[0071] Where N S This represents the number of superpixels.

[0072] After the superpixel segmentation is completed, the i-th superpixel in X and Y are represented as follows: and

[0073] Step 104: Extract the mean and variance of superpixels from the superpixel segmentation results as the features of the superpixels to construct the initial feature matrix; use the KNN algorithm to calculate the initial adjacency matrix of the superpixel segmentation results of the pre-event image and the post-event image.

[0074] By extracting the mean and variance information of each channel within a superpixel as its features, an initial feature matrix is ​​obtained to represent each superpixel. and Each column represents the feature vector of a superpixel.

[0075] After converting the image into a superpixel representation, each superpixel can be viewed as a node in the graph, and its feature vector represents the node's features. This application uses the K-Nearest Neighbors (KNN) algorithm to construct an initial adjacency matrix to capture the structural information of the image before and after the event.

[0076] Step 106: Perform graph structure learning on the initial adjacency matrix and initial feature matrix according to the graph autoencoder to obtain the feature matrix of the hidden layer and the new adjacency matrix; use the structural consistency principle to perform consistency calculation on the feature matrix of the hidden layer and the new adjacency matrix of the image before the event and the image after the event to obtain the backward structural difference vector in the domain before the event and the forward structural difference vector in the domain after the event.

[0077] This paper leverages the powerful graph representation capabilities of Graph Autoencoders (GAEs) to optimize the initial adjacency matrix. First, two graph convolutional layers encode the features of the hidden layer. After enhancing node features through the encoder, a new adjacency matrix is ​​obtained using the decoder. However, the graph structure learning processes before and after events are two independent processes, failing to utilize the fact that the forward and backward structural difference vectors should be consistent in HCD tasks to improve graph structure learning. To address this issue, this application introduces a joint learning method that considers the characteristics of HCD tasks. Specifically, when using GAE to obtain Z... x , Z y and Then, based on the principle of structural consistency, the backward structural difference vector in the pre-event domain and the forward structural difference vector in the post-event domain are calculated.

[0078] Step 108: Train the graph autoencoder according to the pre-set reconstruction loss function, change alignment loss function and sparse loss function to obtain the trained graph autoencoder.

[0079] Since the initial KNN graph can represent the topological relationships between land covers to some extent, in order to make the adjacency matrix learned by GAE consistent with the initial A... x This is relatively similar, achieved by reconstructing the loss function;

[0080] The forward and backward structural difference vectors represent the probability that the same event causes changes in the images of the preceding and following events, and both should be the same. This application designs a change-alignment loss function to constrain the graph structure learning process. This function restricts the forward and backward structural difference vectors to have the same value at the same node, and to be as large as possible. Without constraints on the change-alignment loss function, false alarms may occur. To constrain the change-alignment loss function, this application introduces a sparse loss function to force the forward and backward structural difference vectors to be sparse, such that f... x and f y It will tend towards 0.

[0081] In change detection tasks, the fact that the regions where changes occur typically occupy only a small portion of the entire image makes the structural difference vector sparse. This is addressed by using the change alignment loss function L... sp and sparse loss function L ca The adversarial processes between the graphs jointly optimize the graph structure, resulting in a learned graph structure that exhibits HCD characteristics. In summary, this application utilizes the pre-event image reconstruction loss function L... reconx The image reconstruction loss function L after the event recony L ca and L sp The GAE with common constraints has the following total loss function:

[0082] L all =L reconx +L recony +αL ca +βL sp

[0083] Where α and β are respectively L ca and L sp The weight.

[0084] By setting reconstruction loss function, change alignment loss function, and sparse loss function, prior knowledge that the differences between forward and backward structures should capture the same changing events is incorporated into graph structure learning. The training process of the graph autoencoder is guided by the loss function, which promotes the fusion of differences between forward and backward structures and the optimization of graph structure.

[0085] Step 110: Use the trained graph autoencoder to learn the graph structure of the initial adjacency matrix of the pre-event image and the post-event image, and obtain the optimized backward structure difference vector in the pre-event domain and the forward structure difference vector in the post-event domain.

[0086] In L all Under the joint constraints, GAE can learn node representations and connections between nodes that possess HCD properties. The learned graph structure is then used to compute the final f. x and f y .

[0087] Step 112: Generate a forward difference image and a backward difference image based on the optimized backward structure difference vector in the event front domain and the forward structure difference vector in the event back domain; fuse the forward difference image and the backward difference image to obtain the final difference image; perform Otsu thresholding on the difference image to obtain a binary change image; the binary change image is the result of heterogeneous change detection.

[0088] The difference image is generated by assigning the difference vectors of nodes to each pixel within the corresponding superpixel. Ultimately, the forward difference image DI is obtained. y and backward difference image DI x The fusion produces the final difference image (DI). final .

[0089] After obtaining DI final Subsequently, the HCD task can be viewed as a binary classification problem. To verify the effectiveness of the previous steps, this application employs the simplest thresholding method, the Otsu method, to obtain the final change graph. After obtaining the final change graph, an iterative process is used to backpropagate the changed nodes to the optimized graph structure, removing the influence of the changed nodes and thus improving the final change detection performance.

[0090] In the aforementioned unsupervised heterogeneous change detection method based on graph structure learning, the pre-event and post-event images are first co-segmented using a simple linear iterative clustering algorithm to obtain identical superpixel segmentation results for both images. Then, the mean and variance of the superpixels are extracted as their features to obtain an initial feature matrix. By using superpixels as the basic unit for heterogeneous change detection instead of individual pixels, the algorithm complexity is reduced. Furthermore, the capture of local textures and global structures in the image is improved by utilizing contextual information. Finally, the KNN algorithm is used to construct an initial adjacency matrix to capture the structural information of the pre- and post-event images. The powerful graph representation capabilities of the graph autoencoder are then used to optimize the initial adjacency matrix. By leveraging the ability of graph autoencoders to learn complex structures and relationships in graph data, this method overcomes the limitations of K-nearest neighbor graphs in capturing topological relationships between real land cover, improving the accuracy of image feature capture. Furthermore, by setting reconstruction loss functions, change alignment loss functions, and sparsity loss functions, prior knowledge that forward and backward structural differences should capture the same change events is incorporated into graph structure learning. The training process of the graph autoencoder, guided by the loss functions, promotes the fusion of forward and backward structural differences and the optimization of graph structure, enabling more effective fusion of complementary information between forward and backward structural differences, improving the accuracy of the final difference image, and thus enhancing the accuracy of final change detection.

[0091] In one embodiment, the pre-event image and the post-event image are co-segmented using a simple linear iterative clustering algorithm to obtain the same superpixel segmentation results for the pre-event image and the post-event image, including:

[0092] The pre-event and post-event images are co-segmented using a simple linear iterative clustering algorithm, resulting in identical superpixel segmentation results for both pre-event and post-event images.

[0093]

[0094]

[0095] Where, N S Let be the number of superpixels, and i and j represent different superpixel indices. The i-th superpixel in the image before the event and the image after the event are respectively denoted as . and m represents the ordinate of a pixel, n represents the x-coordinate of a pixel, M represents the height of the image, N represents the width of the image, and C represents the height of the image. X C represents the number of channels in the image before the event. Y This indicates the number of channels in the image after the event.

[0096] In one embodiment, the KNN algorithm is used to calculate the initial adjacency matrix of the pre-event and post-event images based on the superpixel segmentation results, including:

[0097] The initial adjacency matrix of the image before the event is obtained by calculating the superpixel segmentation results of the image before the event using the KNN algorithm.

[0098]

[0099]

[0100] in, Represents and Superpixel X i The set of K most similar superpixels, where the value of K is empirically chosen as... N S This represents the number of superpixels.

[0101] In one embodiment, graph structure learning is performed on the initial adjacency matrix and the initial feature matrix according to the graph autoencoder to obtain the feature matrix of the hidden layer and a new adjacency matrix, including:

[0102] The initial adjacency matrix and initial feature matrix of the image before the event are encoded using graph convolutional layers, resulting in the feature matrix of the hidden layer.

[0103]

[0104] Where σ is the ReLU nonlinear activation function, A x I is the initial adjacency matrix of the image before the event, and I is the identity matrix. yes The degree matrix, This is the weight matrix of the first graph convolutional layer. The initial feature matrix of the image before the event. This is the weight matrix of the second graph convolutional layer;

[0105] The feature matrix is ​​decoded using a decoder to obtain a new adjacency matrix.

[0106]

[0107] Where T represents the transpose operation.

[0108] In a specific embodiment, A is used to construct the image before the event. y Z y and The calculation process is the same as the image before the event.

[0109] In one embodiment, consistency calculations are performed on the feature matrices and new adjacency matrices of the hidden layers of the pre-event and post-event images according to the structural consistency principle, to obtain the backward structural difference vector in the pre-event domain and the forward structural difference vector in the post-event domain, including:

[0110] Based on the structural consistency principle, consistency calculations are performed on the feature matrix of the hidden layer of the pre-event image, the new adjacency matrix of the pre-event image, and the post-event image to obtain the backward structural difference vector in the pre-event domain.

[0111]

[0112] Among them, Γ x It is the distance metric matrix, Γ x Each element in the matrix represents the Euclidean distance between different feature vectors in the feature matrix of the hidden layer of the image before the event. This represents the new adjacency matrix of the image before the event. Represents the new adjacency matrix of the image after the event;

[0113] Based on the structural consistency principle, consistency calculations are performed on the feature matrix of the post-event image, the optimized adjacency matrix of the pre-event image, and the post-event image to obtain the forward structural difference vector f in the post-event domain. y .

[0114] In a specific embodiment, f y The calculation process and f i x similar.

[0115] In one embodiment, the pre-set reconstruction loss function includes a pre-set reconstruction loss function for the image before the event and a pre-set reconstruction loss function for the image after the event; the pre-set reconstruction loss function for the image before the event is...

[0116]

[0117] Where, N S A represents the number of superpixels, where i and j represent the different superpixel indices. x This represents the initial adjacency matrix.

[0118] In one embodiment, the change alignment loss function is:

[0119]

[0120] Among them, f i x f represents the backward structural difference vector corresponding to the i-th superpixel. i yThe forward structure difference vector corresponding to the i-th superpixel

[0121] In one embodiment, the sparse loss function is:

[0122]

[0123] In one embodiment, the forward difference image and the backward difference image are fused to obtain the final difference image, including:

[0124] The forward and backward difference images are fused to obtain the final difference image.

[0125] DI final =(DI) x / max(DI x )+DI y / max(DI y )) / 2

[0126] Among them, DI x Represents the backward difference image, DI y This represents a forward difference image.

[0127] In one embodiment, Otsu thresholding is performed on the difference image to obtain a binary transformation image, including:

[0128] Otsu thresholding is performed on the difference image to obtain a binary transformation image.

[0129]

[0130] Where CM is a binary transformation image, and T is the threshold obtained by the Otsu method.

[0131] In a specific embodiment, after obtaining DI final Subsequently, the HCD task can be viewed as a binary classification problem. To verify the effectiveness of the previous steps, the simplest thresholding method—the Otsu method—is used to obtain the final change graph. After obtaining the final change graph, an iterative process is used to backpropagate the changed nodes to the optimized graph structure, removing the influence of the changed nodes and thus improving the final change detection performance.

[0132] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0133] In one embodiment, such as Figure 2 As shown, an unsupervised heterogeneous change detection device based on graph structure learning is provided, including: a superpixel segmentation module 202, a neighbor graph calculation module 204, a graph structure learning module 206, a graph autoencoder training module 208, a difference vector calculation module 210, and a difference detection module 212, wherein:

[0134] The superpixel segmentation module 202 is used to acquire the pre-event image and the post-event image to be detected; and to perform co-segmentation of the pre-event image and the post-event image according to a simple linear iterative clustering algorithm to obtain the same superpixel segmentation results for the pre-event image and the post-event image.

[0135] The neighbor graph calculation module 204 is used to extract the mean and variance of superpixels from the superpixel segmentation results as the features of the superpixels to construct the initial feature matrix; and to calculate the initial adjacency matrix of the superpixel segmentation results of the pre-event image and the post-event image using the KNN algorithm.

[0136] The graph structure learning module 206 is used to perform graph structure learning on the initial adjacency matrix and initial feature matrix according to the graph autoencoder to obtain the feature matrix of the hidden layer and the new adjacency matrix; and to perform consistency calculation on the feature matrix of the hidden layer and the new adjacency matrix of the image before the event and the image after the event using the structural consistency principle to obtain the backward structural difference vector in the domain before the event and the forward structural difference vector in the domain after the event.

[0137] The graph autoencoder training module 208 is used to train the graph autoencoder according to the pre-set reconstruction loss function, change alignment loss function and sparse loss function to obtain the trained graph autoencoder.

[0138] The difference vector calculation module 210 is used to perform graph structure learning on the initial adjacency matrix of the pre-event image and the post-event image using the trained graph autoencoder, so as to obtain the optimized backward structure difference vector in the pre-event domain and the forward structure difference vector in the post-event domain.

[0139] The difference detection module 212 is used to generate a forward difference image and a backward difference image based on the optimized backward structure difference vector in the event front domain and the forward structure difference vector in the event back domain; to fuse the forward difference image and the backward difference image to obtain the final difference image; to perform Otsu thresholding on the difference image to obtain a binary change image; the binary change image is the result of heterogeneous change detection.

[0140] Specific limitations regarding the unsupervised heterogeneous change detection device based on graph structure learning can be found in the limitations of the unsupervised heterogeneous change detection method based on graph structure learning mentioned above, and will not be repeated here. Each module in the aforementioned unsupervised heterogeneous change detection device based on graph structure learning can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0141] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An unsupervised multi-source change detection method based on graph structure learning, characterized in that, The method comprises: obtaining pre-event images and post-event images to be detected; co-segmenting the pre-event images and the post-event images according to a simple linear iterative clustering algorithm to obtain superpixel segmentation results of the pre-event images and the post-event images; extracting mean variances of superpixels in the superpixel segmentation results as features of the superpixels to construct an initial feature matrix; calculating the superpixel segmentation results of the pre-event images and the post-event images by using a KNN algorithm to obtain an initial adjacency matrix of the pre-event images and the post-event images; performing graph structure learning on the initial adjacency matrix and the initial feature matrix according to a graph autoencoder to obtain a feature matrix of a hidden layer and a new adjacency matrix; and performing consistency calculation on the feature matrix of the hidden layer and the new adjacency matrix of the pre-event images and the post-event images according to a structure consistency principle to obtain a backward structural difference vector in a pre-event domain and a forward structural difference vector in a post-event domain; training the graph autoencoder according to a pre-set reconstruction loss function, a change alignment loss function and a sparsity loss function to obtain a trained graph autoencoder; performing graph structure learning on the initial adjacency matrix of the pre-event images and the post-event images by using the trained graph autoencoder to obtain an optimized backward structural difference vector in the pre-event domain and an optimized forward structural difference vector in the post-event domain; generating a forward difference image and a backward difference image according to the optimized backward structural difference vector in the pre-event domain and the optimized forward structural difference vector in the post-event domain; fusing the forward difference image and the backward difference image to obtain a final difference image; performing Otsu threshold segmentation on the difference image to obtain a binary change image; and 2. The method of claim 1, wherein, co-segmenting the pre-event images and the post-event images according to a simple linear iterative clustering algorithm to obtain superpixel segmentation results of the pre-event images and the post-event images, comprising: co-segmenting the pre-event images and the post-event images according to a simple linear iterative clustering algorithm to obtain superpixel segmentation results of the pre-event images and the post-event images, which is Δ S = {Δ i S i = 1,..., N s} ; wherein N S is the number of superpixels, i and j represent different superpixel indices, the ith superpixel in the pre-event image and the post-event image are denoted as and m represents the vertical coordinate of a pixel, n represents the horizontal coordinate of a pixel, M represents the height of an image, M represents the height of an image, N represents the width of an image, C X represents the number of channels of the pre-event image, C Y represents the number of channels of the post-event image.

3. The method of claim 2, wherein, calculating the superpixel segmentation results of the pre-event images and the post-event images by using a KNN algorithm to obtain an initial adjacency matrix of the pre-event images and the post-event images, comprising: calculating the superpixel segmentation results of the pre-event images by using a KNN algorithm to obtain an initial adjacency matrix of the pre-event images, which is wherein, represent the K most similar superpixels set to superpixel X i most similar superpixels set to superpixel X N S is the number of superpixels.

4. The method according to any one of claims 1 to 3, characterized in that, performing graph structure learning on the initial adjacency matrix and the initial feature matrix according to a graph autoencoder to obtain a feature matrix of a hidden layer and a new adjacency matrix, comprising: encoding the initial adjacency matrix and the initial feature matrix of the pre-event images by using a graph convolution layer to obtain a feature matrix of a hidden layer, which is wherein σ is a relu nonlinear activation function, A x is an initial adjacency matrix of the pre-event image, I is a unit matrix, is a degree matrix of is a weight matrix of the first graph convolution layer, is an initial feature matrix of the pre-event image, is a weight matrix of the second graph convolution layer; decoding the feature matrix by using a decoder to obtain a new adjacency matrix, which is wherein T represents a transposition operation.

5. The method of claim 4, wherein, performing consistency calculation on the feature matrix of the hidden layer and the new adjacency matrix of the pre-event images and the post-event images according to a structure consistency principle to obtain a backward structural difference vector in a pre-event domain and a forward structural difference vector in a post-event domain, comprising: According to the structural consistency principle, the feature matrix of the hidden layer of the pre-event image, and the new adjacency matrix of the pre-event image and the post-event image are calculated to obtain the backward structural difference vector in the pre-event field as wherein Γ x is a distance metric matrix, Γ x each element of which represents the Euclidean distance between different eigenvectors in the feature matrix of the hidden layer of the pre-event image, denotes a new adjacency matrix of the pre-event image, denotes a new adjacency matrix of the post-event image; According to the structural consistency principle, the consistency of the feature matrix of the post-event image, the optimized adjacent matrix of the pre-event image and the post-event image is calculated to obtain the forward structural difference vector f in the post-event field y .

6. The method of claim 5, wherein, The pre-set reconstruction loss function includes a pre-set reconstruction loss function of the pre-event image and a pre-set reconstruction loss function of the post-event image; the pre-set reconstruction loss function of the pre-event image is where N S is the number of superpixels, i and j represent different superpixel indices, A x denotes the initial adjacency matrix.

7. The method of claim 6, wherein, The change alignment loss function is wherein f i x represents the backward structural difference vector corresponding to the i-th superpixel, f i y represents the forward structural difference vector corresponding to the i-th superpixel; The sparse loss function is 8. The method of claim 1, wherein, The forward difference image and the backward difference image are fused to obtain a final difference image, including: The forward difference image and the backward difference image are fused to obtain a final difference image as DI final = (DI x / max(DI x ) + DI y / max(DI y )) / 2 where DI x represents a backward difference image, DI y represents a forward difference image.

9. The method of claim 8, wherein, The Otsu threshold segmentation is performed on the difference image to obtain a binary change image, including: The Otsu threshold segmentation is performed on the difference image to obtain a binary change image as Wherein, CM is the binary change image, and T is a threshold value obtained by the Otsu method.

10. An unsupervised change detection apparatus based on graph structure learning, characterized in that, The device includes: A superpixel segmentation module is configured to acquire a pre-event image and a post-event image to be detected, and perform cooperative segmentation on the pre-event image and the post-event image according to a simple linear iterative clustering algorithm to obtain superpixel segmentation results of the pre-event image and the post-event image; An adjacency graph calculation module is configured to extract mean variance of superpixels in the superpixel segmentation results as features of the superpixels to construct an initial feature matrix, and calculate the superpixel segmentation results of the pre-event image and the post-event image by using a KNN algorithm to obtain an initial adjacency matrix of the pre-event image and the post-event image; A graph structure learning module is configured to perform graph structure learning on the initial adjacency matrix and the initial feature matrix according to a graph autoencoder to obtain a feature matrix of a hidden layer and a new adjacency matrix, and perform consistency calculation on the feature matrix of the hidden layer and the new adjacency matrix of the pre-event image and the post-event image according to a structural consistency principle to obtain a backward structural difference vector in a pre-event field and a forward structural difference vector in a post-event field; A graph autoencoder training module is configured to train the graph autoencoder according to a pre-set reconstruction loss function, a change alignment loss function and a sparse loss function to obtain a trained graph autoencoder; A difference vector calculation module is configured to perform graph structure learning on the initial adjacency matrix of the pre-event image and the post-event image by using the trained graph autoencoder to obtain an optimized backward structural difference vector in the pre-event field and an optimized forward structural difference vector in the post-event field; A difference detection module is configured to generate a forward difference image and a backward difference image according to the optimized backward structural difference vector in the pre-event field and the optimized forward structural difference vector in the post-event field, fuse the forward difference image and the backward difference image to obtain a final difference image, perform Otsu threshold segmentation on the difference image to obtain a binary change image, and the binary change image is a heterogeneous change detection result.

Images