Method and device for change detection of multi-source remote sensing images based on hypergraph neural network

Abstract

The application relates to the technical field of image processing, in particular to a multi-source remote sensing image change detection method and device based on a hypergraph neural network, wherein the method comprises the following steps: constructing nodes based on a given multi-source remote sensing image pair, calculating the correlation between the nodes, obtaining a correlation matrix to form a hyperedge, thereby constructing a local hypergraph and a global hypergraph with nodes and hyperedges; learning by using a hypergraph neural network combined with the correlation matrix, extracting image features before and after changes, and outputting probability prediction results of local and global hypergraph node changes; using a fusion strategy to fuse the probability prediction results of local and global hypergraph node changes, obtaining a final fusion prediction result of each pixel; and finally judging whether each pixel has changed by using a threshold classification method, and obtaining a final change detection result. Therefore, the precision of multi-source heterogeneous image change detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-source remote sensing image change detection method and device based on a hypergraph neural network. BACKGROUND

[0002] In related technologies, remote sensing images play an important role in ground environment monitoring, disaster emergency management and urban and rural development planning, and multi-source heterogeneous images can achieve real-time and effective change monitoring, but compared with single modal double time phase remote sensing data, the change detection precision is low.

[0003] At present, the change detection of multi-source heterogeneous remote sensing images mainly includes methods based on homogeneous transformation and deep feature learning. The method based on homogeneous transformation has poor generalization ability, and when the scene is complex or the noise influence is large, the conversion model between modes constructed by artificial may fail, thereby causing the change detection performance to decline. The method based on deep feature learning needs more data sources for learning, and the multi-source heterogeneous remote sensing images have different data types and less data, which is difficult to meet the training needs of deep neural networks, and improvement is urgently needed. SUMMARY

[0004] The present application provides a multi-source remote sensing image change detection method and device based on a hypergraph neural network to solve the problem of low change detection precision of multi-source heterogeneous images in related technologies.

[0005] The first aspect of the present application provides a multi-source remote sensing image change detection method based on a hypergraph neural network, comprising the following steps: based on a target multi-source remote sensing image pair, constructing a plurality of nodes of the target multi-source remote sensing image pair, and calculating the correlation between the nodes in the plurality of nodes to obtain a first correlation matrix of a local hypergraph and a second correlation matrix of a global hypergraph, respectively, and forming a plurality of hyperedges of the corresponding hypergraph using each correlation matrix to construct a local hypergraph and a global hypergraph with the plurality of nodes and the plurality of hyperedges; based on the first correlation matrix and the second correlation matrix, using a preset hypergraph neural network to perform hypergraph learning to extract the features before and after the change of the multi-source remote sensing image pair, and generate a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph, respectively; fuse the first probability prediction result and the second probability prediction result to obtain the final fusion prediction result of each pixel, and obtain the final change detection result of the target multi-source remote sensing image pair according to the final fusion prediction result of each pixel.

[0006] By the technical solution, the embodiment of the present application can effectively represent the features of the multi-source remote sensing image pair before and after the change by constructing local supergraph and global supergraph, using nodes and their associated relationships, and accurately predicting the probability of node change through the learning of the supergraph neural network. In combination with the fusion strategy, the local and global information is comprehensively considered. Finally, the threshold classification method is used to determine the change of each pixel, so that the detection result is clearer.

[0007] Optionally, in an embodiment of the present application, the constructing the local supergraph and the global supergraph with the plurality of nodes and the plurality of superedges comprises: segmenting the multi-source remote sensing image pair to generate a plurality of image blocks, constructing the local supergraph with the pixels inside each image block of the plurality of image blocks as nodes, and constructing the global supergraph with each image block of the plurality of image blocks as nodes.

[0008] By the technical solution, the embodiment of the present application can effectively capture the local features and global information in the image by segmenting the multi-source remote sensing image into a plurality of image blocks, constructing the local supergraph with the pixels inside the image block as nodes, and constructing the global supergraph with the entire image block as nodes. This hierarchical supergraph structure not only improves the accuracy of change detection, but also enhances the sensitivity to the feature differences between different modal images, so as to more accurately identify the changed area in complex scenes and improve the overall performance of remote sensing image change detection.

[0009] Optionally, in an embodiment of the present application, the constructing the plurality of nodes of the target multi-source remote sensing image pair and calculating the associated relationship between the nodes in the plurality of nodes based on the target multi-source remote sensing image pair to respectively obtain a first associated matrix of the local supergraph and a second associated matrix of the global supergraph comprises:

[0010] The calculation formula of the first associated matrix is:

[0011]

[0012] dis_Eu i is the Euclidean distance between the current node and the i-th node, and dis_avg_E is the average Euclidean distance between the current node and all other nodes except the current node.

[0013] Based on the spatial position of each image block in the unsegmented image, the spatial distance between the current node and a plurality of spatially adjacent position nodes around the current node is calculated by using a spatial position calculation method, and the calculation formula of the spatial distance is:

[0014]

[0015] wherein x i and yi is a two-dimensional spatial coordinate of the current node, x j and y j is a two-dimensional spatial coordinate of a plurality of spatially adjacent position nodes around the current node;

[0016] The second correlation matrix is obtained by using the spatial distance, and a calculation formula of the second correlation matrix is:

[0017]

[0018] wherein dis_Su i is a spatial distance between the current node and the node i, and dis_avg_s is an average spatial distance between the current node and the remaining neighborhood nodes except the current node.

[0019] By the technical solution, the embodiments of the present application can effectively capture the correlation between nodes by constructing the correlation matrix of local and global hypergraphs, using Euclidean distance and spatial distance. The correlation matrix of the local hypergraph is calculated by the Euclidean distance between the target node and other nodes, which can accurately reflect the structural characteristics within the image block. The correlation matrix of the global hypergraph is calculated by the spatial position, which considers the position relationship of the image block in the whole image, and enhances the understanding of the spatial characteristics. The advantage of this method is that it can make full use of the characteristics of multi-source heterogeneous remote sensing images, improve the accuracy and robustness of change detection, and thus achieve more accurate change detection results in complex scenes.

[0020] Optionally, in an embodiment of the present application, the hypergraph neural network comprises a feature extraction module, an aggregation network module and a classification module, wherein the hypergraph learning based on the first correlation matrix and the second correlation matrix is performed by using a preset hypergraph neural network to extract the features of the multi-source remote sensing image pair before and after the change, and to generate the first probability prediction result of the node change of the local hypergraph and the second probability prediction result of the node change of the global hypergraph, respectively, including: inputting the plurality of image blocks and the first correlation matrix into the preset hypergraph neural network to perform the local hypergraph learning, and extracting the local hypergraph features of the plurality of image blocks before and after the change by the feature extraction module, and processing the local hypergraph features by the aggregation network module to obtain processed local hypergraph features, and classifying the processed local hypergraph features by the classification module to obtain the first probability prediction result; inputting the node data composed of all the center pixels of the plurality of image blocks and the second correlation matrix into the preset hypergraph neural network to perform the global hypergraph learning, and extracting the global hypergraph features of the multi-source remote sensing image pair before and after the change by the feature extraction module, and processing the global hypergraph features by the aggregation network module to obtain processed global hypergraph features, and classifying the processed global hypergraph features by the classification module to obtain the second probability prediction result.

[0021] Through the above technical solution, the embodiments of the present application can effectively extract and process the local and global hypergraph features of the multi-source remote sensing image through the cooperative work of the feature extraction module, the aggregation network module and the classification module in the hypergraph neural network, and then accurately predict the probability of node change. This structure not only improves the ability to capture image features before and after the change, but also enhances the accuracy of change detection by aggregating information at different levels, significantly improving the precision of multi-source heterogeneous remote sensing image change detection.

[0022] The second aspect embodiment of the application provides a multi-source remote sensing image change detection device based on a hypergraph neural network, comprising: a hypergraph construction module, configured to construct a plurality of nodes of a target multi-source remote sensing image pair based on the target multi-source remote sensing image pair, calculate the correlation between nodes in the plurality of nodes, to obtain a first correlation matrix of a local hypergraph and a second correlation matrix of a global hypergraph respectively, and form a plurality of hyperedges of the corresponding hypergraph by using each correlation matrix, to construct the local hypergraph and the global hypergraph with the plurality of nodes and the plurality of hyperedges; a hypergraph learning module, configured to perform hypergraph learning by using a preset hypergraph neural network based on the first correlation matrix and the second correlation matrix, to extract features before and after changes of the multi-source remote sensing image pair, and generate a first probability prediction result of node changes of the local hypergraph and a second probability prediction result of node changes of the global hypergraph respectively; and a fusion prediction module, configured to fuse the first probability prediction result and the second probability prediction result to obtain a final fusion prediction result of each pixel, and obtain a final change detection result of the target multi-source remote sensing image pair according to the final fusion prediction result of each pixel.

[0023] Through the above technical solution, the embodiment of the application can effectively represent the features before and after changes of the multi-source remote sensing image pair by constructing the local hypergraph and the global hypergraph, using the nodes and their correlation, accurately predict the probability of node changes through the learning of the hypergraph neural network, and finally determine the change of each pixel by using the threshold classification method, so that the detection result is clearer.

[0024] Optionally, in an embodiment of the application, the hypergraph construction module comprises a segmentation unit configured to segment the multi-source remote sensing image pair to generate a plurality of image blocks, construct the local hypergraph by using the pixels inside each image block of the plurality of image blocks as nodes, and construct the global hypergraph by using each image block of the plurality of image blocks as nodes.

[0025] Through the above technical solution, the embodiment of the application can effectively capture the local features and global information in the image by segmenting the multi-source remote sensing image into a plurality of image blocks, constructing the local hypergraph by using the pixels inside the image block as nodes, and constructing the global hypergraph by using the entire image block as nodes. This hierarchical hypergraph structure not only improves the accuracy of change detection, but also enhances the sensitivity to the feature differences between different modal images, so as to more accurately identify the changed area in a complex scene and improve the overall performance of remote sensing image change detection.

[0026] Optionally, in an embodiment of the application, the hypergraph construction module comprises a local hypergraph construction unit configured to obtain the first correlation matrix of the local hypergraph, and the calculation formula of the first correlation matrix is:

[0027]

[0028] dis_Eu i is the Euclidean distance between the current node and the i-th node, and dis_avg_E is the average Euclidean distance between the current node and all other nodes except the current node.

[0029] The calculation unit is configured to calculate, based on the spatial position of each image block in the unsegmented image, the spatial distance between the current node and a plurality of spatially adjacent position nodes around the current node by taking the current node as a center point and using a spatial position calculation method, and the calculation formula of the spatial distance is:

[0030]

[0031] wherein x i and y i are the two-dimensional spatial coordinates of the current node, and x j and y j are the two-dimensional spatial coordinates of the plurality of spatially adjacent position nodes around the current node.

[0032] The global hypergraph construction unit is configured to obtain the second correlation matrix by using the spatial distance, and the calculation formula of the second correlation matrix is:

[0033]

[0034] dis_Su i is the spatial distance between the current node and the i-th node, and dis_avg_S is the average spatial distance between the current node and the remaining neighborhood nodes except the current node.

[0035] Through the above technical solutions, the embodiments of the present application can effectively capture the correlation between nodes by constructing the correlation matrix of local and global hypergraphs using Euclidean distance and spatial distance. The correlation matrix of the local hypergraph is calculated by the Euclidean distance between the target node and other nodes, which can accurately reflect the structural characteristics within the image block. The correlation matrix of the global hypergraph is calculated by the spatial position, which considers the position relationship of the image block in the whole image, and enhances the understanding of the spatial features. The advantage of this method is that it can fully utilize the characteristics of multi-source heterogeneous remote sensing images, improve the accuracy and robustness of change detection, and thus achieve more accurate change detection results in complex scenes.

[0036] Optionally, in an embodiment of the present application, the supergraph neural network comprises a feature extraction module, an aggregation network module and a classification module, wherein the supergraph learning module comprises: a local supergraph learning unit configured to input the plurality of image blocks and the first correlation matrix into the preset supergraph neural network to perform the local supergraph learning, extract local supergraph features of the plurality of image blocks before and after the change through the feature extraction module respectively, process the local supergraph features through the aggregation network module to obtain processed local supergraph features, and classify the processed local supergraph features through the classification module to obtain the first probability prediction result; and a global supergraph learning unit configured to input node data composed of center pixels of all the plurality of image blocks and the second correlation matrix into the preset supergraph neural network to perform the global supergraph learning, extract global supergraph features of the multi-source remote sensing image before and after the change through the feature extraction module respectively, process the global supergraph features through the aggregation network module to obtain processed global supergraph features, and classify the processed global supergraph features through the classification module to obtain the second probability prediction result.

[0037] Through the above technical solution, the embodiments of the present application can effectively extract and process the local and global supergraph features of the multi-source remote sensing image through the collaborative work of the feature extraction module, the aggregation network module and the classification module in the supergraph neural network, and then accurately predict the probability of node change. This structure not only improves the ability to capture image features before and after the change, but also enhances the accuracy of change detection by aggregating information at different levels, significantly improving the precision of multi-source heterogeneous remote sensing image change detection.

[0038] The third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-source remote sensing image change detection method based on the supergraph neural network as described in the above embodiments.

[0039] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the multi-source remote sensing image change detection method based on the supergraph neural network as described above.

[0040] The fifth aspect of the present application provides a computer program product comprising a computer program, which is executed to implement the multi-source remote sensing image change detection method based on the supergraph neural network as described above.

[0041] The embodiments of the present application can effectively extract the features of multi-source remote sensing images before and after the change by constructing local and global hypergraphs, using nodes and their associated relationships, so as to realize high-precision change detection. The local hypergraph is constructed by the pixels inside the image block, which accurately reflects the structural features of the image block. The global hypergraph takes the entire image block as a node, considers the spatial position relationship, and enhances the understanding of spatial features. The feature extraction, aggregation and classification modules of the hypergraph neural network work collaboratively to improve the ability to capture the features of the images before and after the change, and through the fusion of local and global information, the reliability and accuracy of the change detection are improved. Finally, the threshold classification method is used to judge the change of each pixel, so that the detection result is more clear. This method makes full use of the characteristics of multi-source heterogeneous remote sensing images, realizes more accurate change detection in complex scenes, and significantly improves the detection accuracy.

[0042] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, wherein:

[0044] Figure 1 A flowchart of a multi-source remote sensing image change detection method based on a hypergraph neural network according to an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a hypergraph neural network structure according to an embodiment of the present application;

[0046] Figure 3 A public data set used for realizing change detection according to a specific embodiment of the present application;

[0047] Figure 4 A comparison diagram of change detection results and results of other detection methods according to a specific embodiment of the present application;

[0048] Figure 5 A structural schematic diagram of a multi-source remote sensing image change detection device based on a hypergraph neural network according to an embodiment of the present application;

[0049] Figure 6 A structural example diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] Embodiments of the present application are described below in detail with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations are used to denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0051] A hypergraph neural network-based multi-source remote sensing image change detection method and device of an embodiment of the present application is described below with reference to the accompanying drawings. In view of the low detection accuracy of multi-source heterogeneous image change detection in the related art mentioned in the background art, the present application provides a hypergraph neural network-based multi-source remote sensing image change detection method. In this method, the features of multi-source remote sensing images before and after the change can be effectively represented by constructing local hypergraphs and global hypergraphs, and the probability of node change can be accurately predicted through the learning of the hypergraph neural network. In combination with a fusion strategy, local and global information is integrated. Finally, threshold classification is used to determine the change of each pixel, making the detection result clearer. Thus, the problem of low multi-source heterogeneous image change detection accuracy in the related art is solved.

[0052] Specifically, Figure 1 A flowchart of a hypergraph neural network-based multi-source remote sensing image change detection method provided by an embodiment of the present application is shown in FIG. 1.

[0053] As Figure 1 shown, the hypergraph neural network-based multi-source remote sensing image change detection method includes the following steps:

[0054] In step S101, based on a target multi-source remote sensing image pair, a plurality of nodes of the target multi-source remote sensing image pair are constructed, and the association relationship between the nodes in the plurality of nodes is calculated to obtain a first association matrix of a local hypergraph and a second association matrix of a global hypergraph, respectively. A plurality of hyperedges of the corresponding hypergraph are formed by using each association matrix to construct a local hypergraph and a global hypergraph having a plurality of nodes and a plurality of hyperedges.

[0055] It can be understood that the process of hypergraph construction is actually to convert the multi-source remote sensing image into a hypergraph structure having nodes and hyperedges. The nodes are usually pixel points in the image, and the hyperedges represent the association relationship between the image pixels.

[0056] Optionally, in an embodiment of the present application, constructing a local hypergraph and a global hypergraph having a plurality of nodes and a plurality of hyperedges includes: segmenting the multi-source remote sensing image pair to generate a plurality of image blocks, constructing the local hypergraph with the pixels inside each image block of the plurality of image blocks as nodes, and constructing the global hypergraph with each image block of the plurality of image blocks as nodes.

[0057] In actual implementation, the image segmentation method can solve the problem of large remote sensing image size for detecting changes. The image is divided into p×p image blocks, and then local and global hypergraphs are constructed hierarchically using the pixels inside the image blocks and the image blocks themselves as nodes.

[0058] On one hand, the local hypergraph is constructed by using pixels within an image patch as nodes, and the association relationships of hyperedges are measured using Euclidean distance. Specifically, for a target node, the Euclidean distances between that node and all other nodes in the image patch are calculated, and then the K nearest nodes are taken as nodes associated with that target node on a hyperedge. To obtain the association matrix of the local hypergraph, the association weights are calculated using the following formula:

[0059]

[0060] Among them, dis_Eu i Let be the Euclidean distance between the target node and the i-th node, and let dis_avg_E be the average Euclidean distance between the target node and all other nodes in the image patch. Then, the weights of the K nodes closest to the target node in Euclidean distance are stored in a sparse matrix, where the elements on the main diagonal are 1, indicating that each node has the highest correlation with itself, generating the correlation matrix H of the local hypergraph. local .

[0061] On the other hand, when constructing the global hypergraph, the entire image patch is treated as a node. Specifically, in this embodiment, it can be assumed that under a certain size p (p<15), the difference between the center pixel and its surrounding neighboring pixels is small, and it can represent the entire image patch. Therefore, the center pixel is used as the node of the global hypergraph, and the probability value of the predicted node change is assigned to the entire image patch. The hyperedge association relationship of the global hypergraph is mainly based on the spatial position relationship of each node. Specifically, the spatial position of each image patch in the unsegmented image is maintained, and then the distance between the target node position and its eight spatially adjacent nodes is calculated using the target node position as the center point. For the target node j, the distance between it and node i is calculated according to the spatial position, as follows:

[0062]

[0063] Where, x i and y i Let x be the two-dimensional spatial coordinates of the target node. j and y j Let be the two-dimensional spatial coordinates of multiple neighboring nodes. Further, the association weights of the global hypergraph are obtained using spatial distance, calculated as follows:

[0064]

[0065] Among them, dis_Su i Let be the spatial distance between the target node and node i, and let dis_avg_S be the average spatial distance between the target node and its other neighboring nodes. Similarly, the weights between the eight spatially adjacent nodes are stored as a sparse matrix, with the main diagonal elements still set to 1, generating the global hypergraph association matrix H. global .

[0066] This application's embodiments can effectively transform image data into a hypergraph form by constructing nodes and calculating the relationships between nodes on a given pair of multi-source remote sensing images, thus forming a hyperedge structure of local and global hypergraphs. Specifically, the local hypergraph uses pixels within image patches as nodes and uses Euclidean distance to measure the relationships between nodes, generating a local hypergraph association matrix; the global hypergraph uses the entire image patch as a node, using the central pixel to represent the entire image patch, and calculates association weights based on spatial positional relationships to generate a global hypergraph association matrix. This hierarchical hypergraph construction method can not only handle the complexity of large-size remote sensing images but also improve the accuracy of change detection.

[0067] In step S102, based on the first and second correlation matrices, a preset hypergraph neural network is used to perform hypergraph learning to extract features of multi-source remote sensing image pairs before and after changes, and to generate a first probability prediction result of node changes in the local hypergraph and a second probability prediction result of node changes in the global hypergraph, respectively.

[0068] In this embodiment, the update formula for the hypergraph neural network is as follows:

[0069]

[0070] Where H represents the hypergraph incidence matrix; D e and D v Let X and Y represent the diagonal matrices formed by the hyperedges and degrees of the nodes of the hypergraph, respectively; W is the weight matrix of the hypergraph, initialized as an identity matrix; θ represents the network parameters of the hypergraph neural network; X is the node data or hypergraph features input to the hypergraph convolutional layer; and Y is the hypergraph feature obtained by passing X through a hypergraph convolutional layer.

[0071] Optionally, in one embodiment of this application, the hypergraph neural network includes a feature extraction module, an aggregation network module, and a classification module. Based on a first and a second correlation matrix, a preset hypergraph neural network is used to perform hypergraph learning to extract features of multi-source remote sensing images before and after changes, and to generate a first probability prediction result for node changes in the local hypergraph and a second probability prediction result for node changes in the global hypergraph, respectively. This includes: inputting multiple image patches and the first correlation matrix into the preset hypergraph neural network for local hypergraph learning, and extracting local hypergraph features of the multiple image patches before and after changes using the feature extraction module. The system first obtains local hypergraph features by processing local hypergraph features through an aggregation network module, and then classifies these local hypergraph features using a classification module to obtain a first probability prediction result. Next, node data consisting of the center pixels of all multiple image patches and a second correlation matrix are input into a preset hypergraph neural network for global hypergraph learning. A feature extraction module extracts global hypergraph features from multi-source remote sensing images before and after the change, and the aggregation network module processes these global hypergraph features to obtain all processed hypergraph features. Finally, a classification module classifies these processed global hypergraph features to obtain a second probability prediction result.

[0072] Specifically, this application constructs a network structure in which two hypergraph neural networks extract hypergraph features from dual-temporal remote sensing images and then perform aggregation processing, such as... Figure 2 As shown. First, for local hypergraph learning, the image patch node data groups from dual-temporal image segmentation and the hypergraph association matrix group constructed from each image patch are input into two-way hypergraph neural networks. After several layers of hypergraph convolution and non-linear activation layers to extract features, the hypergraph features of the images before and after the change are obtained. Then, the extracted hypergraph features are processed through an aggregation network module, and finally, a classification module is used to predict the probability of node changes. The loss function for local hypergraph learning is the focus loss, and its formula is as follows:

[0073] Focal_loss(p t )=-α t (1-p t ) γ log(p t (5)

[0074] Where, p t The probability of being a positive class, in this article, refers to the probability of change, α. t For class balancing weights, γ is a hyperparameter that adjusts the weights of simpler samples that are reduced.

[0075] The global hypergraph learning adopts the same network, and the node data composed of the center pixels of the image blocks of the two-phase remote sensing images and the obtained global hypergraph correlation matrix are respectively input into the two networks. The network loss function used for training the global hypergraph learning is a binary cross-entropy loss, and the formula is as follows:

[0076]

[0077] wherein N represents the number of nodes, y i represents the category to which the i-th sample belongs, p i represents the predicted probability value of the i-th sample.

[0078] The embodiments of the present application can effectively extract the features corresponding to the multi-source remote sensing images before and after the change by using the hypergraph neural network and combining the hypergraph correlation matrix for hypergraph learning, and output the probability prediction results of the local and global hypergraph node changes. The method constructs a network structure composed of two hypergraph neural networks, respectively extracts, aggregates and classifies the features of the local and global hypergraph, and improves the accuracy of change detection. By using the focal loss and the binary cross-entropy loss as the loss function, the robustness of the model in processing unbalanced samples is further enhanced. This method not only improves the accuracy of change detection, but also effectively deals with the complexity of multi-source heterogeneous remote sensing images.

[0079] In step S103, the first probability prediction result and the second probability prediction result are fused to obtain the final fusion prediction result of each pixel, and the final change detection result of the target multi-source remote sensing image pair is obtained according to the final fusion prediction result of each pixel.

[0080] Specifically, through the training and learning of the hypergraph neural network, the local and global hypergraph node prediction results are obtained. The local nodes are the pixels of the image blocks, and the predicted results can be directly spliced; the global hypergraph predicts the results of the center pixels of the image blocks, which are assigned to the entire image block and then spliced with all image blocks. Finally, the local and global change prediction results are fused using a fusion method. After fusion, threshold classification method is used to determine whether each pixel has changed to obtain the final change detection result.

[0081] The embodiments of the present application can fully utilize the information at different levels by fusing the change probability prediction results of the local and global hypergraph nodes, and improve the accuracy of change detection. The local hypergraph provides detailed pixel-level change information, while the global hypergraph considers the overall features of the image blocks, and the combination of the two makes the detection result more comprehensive and accurate. In addition, threshold classification method is used for final judgment, which simplifies the result processing process and ensures the efficiency and practicability of change detection.

[0082] The present application will be described in detail below with one specific embodiment.

[0083] 1) Input two pairs of heterogeneous remote sensing images, which are taken by different sensors at different times in the same area, and there is actual change between the two images. This embodiment uses the data set shown in Figure 3 , which is a flood change detection data set in the Gloucester area of the United Kingdom. The pre-change image is a synthetic aperture radar image, and the post-change image is a normalized vegetation index image. This image pair is a multi-source heterogeneous remote sensing image pair with changes in the same area at different times.

[0084] 2) Segment the image pair. In this embodiment, the segmentation size is 7x7, so there are 49 pixels inside each image block. If the size of the original image cannot be divided evenly, 0 padding is performed on the right and bottom of the image.

[0085] 3) Construction of local hypergraph: The pixels in each image block are regarded as hypergraph nodes, the Euclidean distance between each node and the remaining nodes is calculated, and then the nearest K nodes are obtained as the nodes on a hyperedge according to formula (1). In this embodiment, K is 10, and the local hypergraph correlation matrix of each image block is obtained.

[0086] 4) Construction of global hypergraph: The center pixel of each image block is regarded as a node, and the spatial distance between the target node and the surrounding 8-neighborhood is calculated according to formula (2) to obtain the global hypergraph correlation matrix of the entire image.

[0087] 5) Select a certain proportion of positive and negative sample image blocks as the training set. In this embodiment, 100 positive and negative samples are selected. The hypergraph data nodes and hypergraph correlation matrices constructed from the two time-phase images are input into the hypergraph neural network shown in Figure 2 , and after training and learning, the node change probability values predicted by the local and global hypergraphs are obtained.

[0088] 6) Use a fusion strategy or method to fuse the change probability values obtained locally and globally. In this embodiment, the fusion strategy uses the DS (Dempster-Shafer, Dempster-Shafer) evidence theory, in which the prediction results of the hypergraph neural network for the local and global are the belief functions of the change and non-change of the pixel point. Assuming that A represents the change, m1 and m2 represent the two quality functions obtained by the local and global hypergraphs, respectively, and represent the change probability values predicted by the local and global hypergraphs, respectively. According to the DS combination rule, the probability of change after fusion is:

[0089]

[0090] 7) According to the prediction probability value after fusion, the threshold segmentation method is used to obtain the final change detection binary result image. As shown inFigure 4 As shown in the figure, the first image is the true value, the second image is the detection result of the HPT (Homogeneous Transformation) method, the third image is the detection result of the SCCN (Spatial Contextual Convolutional Network) method, and the fourth image is the detection result of the method of the application, wherein the HPT is a representative method based on homogeneous transformation, and the SCCN is a representative method based on deep feature learning.

[0091] Based on the experimental results, it is shown that the multi-source remote sensing image change detection method based on the hypergraph neural network can obtain good detection effect and effectively improve the detection precision compared with other methods.

[0092] To further prove the effect of the application, Table 1 shows the numerical comparison results of the method used in the application and other methods. The calculation formula of each numerical evaluation index and Table 1 are as follows:

[0093]

[0094] Table 1

[0095] Method OA F1 KC HPT 96.15 82.48 80.34 SCCN 95.22 77.65 75.02 The method of the present application 98.63 94.34 93.57

[0096] As can be seen from Table 1, consistent with the visual result, the multi-source remote sensing image change detection method based on the hypergraph neural network obtains the optimal detection index performance, and can effectively improve the precision of multi-source heterogeneous remote sensing image change detection.

[0097] According to the multi-source remote sensing image change detection method based on the hypergraph neural network, the local hypergraph and the global hypergraph are constructed, the features of the multi-source remote sensing images before and after the change are effectively represented by using the nodes and their associated relationships, the probability of node change is accurately predicted by using the learning of the hypergraph neural network, the local and global information is comprehensively combined by using the fusion strategy, and finally, the threshold classification method is used to judge the change of each pixel, so that the detection result is more clear.

[0098] Secondly, the multi-source remote sensing image change detection device based on the hypergraph neural network according to the embodiment of the application is described with reference to the accompanying drawings.

[0099] Figure 5 is a block schematic diagram of the multi-source remote sensing image change detection device based on the hypergraph neural network according to the embodiment of the application.

[0100] As Figure 5As shown, the multi-source remote sensing image change detection device 10 based on the hypergraph neural network includes a hypergraph construction module 100, a hypergraph learning module 200, and a fusion prediction module 300.

[0101] Specifically, the hypergraph construction module 100 is configured to construct a plurality of nodes of a target multi-source remote sensing image pair based on the target multi-source remote sensing image pair, and calculate a correlation relationship between nodes in the plurality of nodes to obtain a first correlation matrix of a local hypergraph and a second correlation matrix of a global hypergraph, respectively, and form a plurality of hyperedges of a corresponding hypergraph by using each correlation matrix, so as to construct the local hypergraph and the global hypergraph with the plurality of nodes and the plurality of hyperedges.

[0102] The hypergraph learning module 200 is configured to perform hypergraph learning by using a preset hypergraph neural network based on the first correlation matrix and the second correlation matrix, so as to extract features of the multi-source remote sensing image pair before and after a change, and generate a first probability prediction result of node change of the local hypergraph and a second probability prediction result of node change of the global hypergraph, respectively.

[0103] The fusion prediction module 300 is configured to fuse the first probability prediction result and the second probability prediction result to obtain a final fusion prediction result of each pixel, and obtain a final change detection result of the target multi-source remote sensing image pair according to the final fusion prediction result of each pixel.

[0104] Optionally, in an embodiment of the present application, the hypergraph construction module 100 includes a segmentation unit configured to segment the multi-source remote sensing image pair to generate a plurality of image blocks, construct a local hypergraph by using pixels inside each image block of the plurality of image blocks as nodes, and construct a global hypergraph by using each image block of the plurality of image blocks as nodes.

[0105] Optionally, in an embodiment of the present application, the hypergraph construction module 100 includes a local hypergraph construction unit, a calculation unit, and a global hypergraph construction unit.

[0106] The local hypergraph construction unit is configured to obtain the first correlation matrix of the local hypergraph, and a calculation formula of the first correlation matrix is as follows:

[0107]

[0108] dis_Eu i dis_avg_E is an average Euclidean distance between the current node and all other nodes except the current node.

[0109] The computing unit is configured to calculate, based on a spatial position of each image block in the unsegmented image, a spatial distance between the current node and a plurality of spatial neighboring position nodes around the current node with the current node as a center point, by using a spatial position calculation method, and a calculation formula of the spatial distance is:

[0110]

[0111] wherein x i and y i are two-dimensional spatial coordinates of the current node, and x j and y j are two-dimensional spatial coordinates of the plurality of spatial neighboring position nodes around the current node.

[0112] The global hypergraph construction unit is configured to obtain a second correlation matrix by using the spatial distance, and a calculation formula of the second correlation matrix is:

[0113]

[0114] wherein dis_Su i is the spatial distance between the current node and the node i, and dis_avg_S is an average spatial distance between the current node and the remaining neighborhood nodes except the current node.

[0115] Optionally, in an embodiment of the present application, the hypergraph neural network comprises a feature extraction module, an aggregation network module and a classification module, wherein the hypergraph learning module 200 comprises a local hypergraph learning unit and a global hypergraph learning unit.

[0116] The local hypergraph learning unit is configured to input the plurality of image blocks and the first correlation matrix into a preset hypergraph neural network for local hypergraph learning, extract local hypergraph features of the plurality of image blocks before and after the change by the feature extraction module respectively, process the local hypergraph features by the aggregation network module to obtain processed local hypergraph features, and classify the processed local hypergraph features by the classification module to obtain a first probability prediction result.

[0117] The global hypergraph learning unit is configured to input node data composed of center pixels of all the plurality of image blocks and the second correlation matrix into a preset hypergraph neural network for global hypergraph learning, extract global hypergraph features of the multi-source remote sensing image before and after the change by the feature extraction module respectively, process the global hypergraph features by the aggregation network module to obtain processed global hypergraph features, and classify the processed global hypergraph features by the classification module to obtain a second probability prediction result.

[0118] It should be noted that the foregoing explanation and description of the embodiment of the method for change detection of multi-source remote sensing images based on a hypergraph neural network also apply to the embodiment of the device for change detection of multi-source remote sensing images based on a hypergraph neural network, which will not be described here again.

[0119] The device for change detection of multi-source remote sensing images based on a hypergraph neural network according to the embodiment of the present application can effectively represent the features of multi-source remote sensing images before and after changes by constructing local hypergraphs and global hypergraphs, using nodes and their associated relationships, and accurately predicting the probability of node changes through the learning of a hypergraph neural network. In combination with a fusion strategy, local and global information is comprehensively used. Finally, a threshold classification method is used to determine the change of each pixel, so that the detection result is clearer.

[0120] Figure 6 The structure schematic diagram of an electronic device is provided for the embodiment of the present application. The electronic device can include:

[0121] The memory 601, the processor 602 and the computer program stored in the memory 601 and executable on the processor 602.

[0122] The processor 602 implements the method for change detection of multi-source remote sensing images based on a hypergraph neural network provided in the above embodiments when executing the program.

[0123] Further, the electronic device further includes:

[0124] The communication interface 603 is used for communication between the memory 601 and the processor 602.

[0125] The memory 601 is used to store the computer program executable on the processor 602.

[0126] The memory 601 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0127] If the memory 601, the processor 602 and the communication interface 603 are independently implemented, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and complete the communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation,Figure 6 Only one bus or bus type is used in some of the drawings to illustrate the architecture, the bus can be replaced by any other type of communications link, such as a digital network.

[0128] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can complete the communication among each other through an internal interface.

[0129] The processor 602 can be a central processing unit (CPU) or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0130] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the hypergraph neural network based multi-source remote sensing image change detection method.

[0131] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed to implement the hypergraph neural network based multi-source remote sensing image change detection method.

[0132] In the description of the present application, the description of the terms “one embodiment”, “some embodiments”, “an example”, “a specific example” or “some examples” means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0133] In addition, the terms “first”, “second” are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first”, “second” can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of “N” is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0134] Any processes or methods described in the flowcharts or elsewhere herein can be understood as representing a module, segment, or portion of code that includes one or N steps for implementing the specified logical functions or processes. The scope of a preferred embodiment of the present application encompasses combinations with additional steps as appropriate, and the steps performed in the manner described or otherwise discussed herein, including in an order other than as described, including in an order substantially simultaneous with one another, or in the reverse order, depending upon the functionality involved. These and other variations are considered to be within the scope of the present application.

[0135] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a product of the manufacturing and / or processing, and can be a machine-readable storage medium (alternatively, the medium can be a machine-readable signal medium). The computer-readable medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Note that the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0136] It should be understood that aspects of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, and in another embodiment, the hardware can be implemented using any or a combination of the following technologies, which are each well known in the art: a discrete logic circuit having logic gates for implementing logic functions upon data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0137] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0138] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0139] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A multi-source remote sensing image change detection method based on hypergraph neural network, characterized in that, The method comprises the following steps: Based on the target multi-source remote sensing image pair, a plurality of nodes of the target multi-source remote sensing image pair are constructed, and the association relationship between nodes in the plurality of nodes is calculated to obtain a first association matrix of a local hypergraph and a second association matrix of a global hypergraph, respectively, and a plurality of hyperedges of the corresponding hypergraph are formed by using each association matrix to construct the local hypergraph and the global hypergraph with the plurality of nodes and the plurality of hyperedges; Based on the first association matrix and the second association matrix, hypergraph learning is performed by using a preset hypergraph neural network to extract features before and after changes of the multi-source remote sensing image pair, and a first probability prediction result of node changes of the local hypergraph and a second probability prediction result of node changes of the global hypergraph are generated, respectively; The first probability prediction result and the second probability prediction result are fused to obtain a final fusion prediction result of each pixel, and a final change detection result of the target multi-source remote sensing image pair is obtained according to the final fusion prediction result of each pixel.

2. The method of claim 1, wherein, The method for constructing the local hypergraph and the global hypergraph with the plurality of nodes and the plurality of hyperedges comprises: The multi-source remote sensing image pair is segmented to generate a plurality of image blocks, and each image block inside the plurality of image blocks is taken as a node to construct the local hypergraph, and each image block of the plurality of image blocks is taken as a node to construct the global hypergraph.

3. The method of claim 2, wherein, The method for constructing the local hypergraph and the global hypergraph with the plurality of nodes and the plurality of hyperedges comprises: The calculation formula of the first association matrix is: wherein, is the Euclidean distance between the current node and the jth node, is the average Euclidean distance between the current node and all other nodes except the current node, is the average Euclidean distance between the current node and all other nodes except the current node. Based on the spatial position of each image block in the unsegmented image, the spatial distance between the current node and a plurality of spatially adjacent position nodes around the current node is calculated by using a spatial position calculation method with the current node as a center point, and the calculation formula of the spatial distance is: wherein, and are two-dimensional spatial coordinates of the current node, and are two-dimensional spatial coordinates of a plurality of spatially adjacent location nodes surrounding the current node. The second association matrix is obtained by using the spatial distance, and the calculation formula of the second association matrix is: in, For the current node and the node Spatial distance, The average spatial distance between the current node and all other neighboring nodes.

4. The method of claim 2, wherein, The hypergraph neural network comprises a feature extraction module, an aggregation network module and a classification module, wherein the method for performing hypergraph learning by using the preset hypergraph neural network based on the first association matrix and the second association matrix to extract features before and after changes of the multi-source remote sensing image pair and generate the first probability prediction result of node changes of the local hypergraph and the second probability prediction result of node changes of the global hypergraph comprises: The plurality of image blocks and the first association matrix are input into the preset hypergraph neural network to perform the local hypergraph learning, and the local hypergraph features of the plurality of image blocks before and after changes are extracted by the feature extraction module, respectively, the local hypergraph features are processed by the aggregation network module to obtain processed local hypergraph features, and the classification module is used to classify the processed local hypergraph features to obtain the first probability prediction result. The node data composed of all the center pixels of the plurality of image blocks is input into the preset hypergraph neural network with the second correlation matrix to perform the global hypergraph learning, and the feature extraction module is used to extract the global hypergraph features of the plurality of source remote sensing image pairs before and after the change respectively, the global hypergraph features are processed by the aggregation network module to obtain processed global hypergraph features, and the classification module is used to classify the processed global hypergraph features to obtain the second probability prediction result.

5. A hypergraph neural network based multi-source remote sensing image change detection apparatus, characterized in that, Comprise: The hypergraph construction module constructs a plurality of nodes of the target multi-source remote sensing image pair based on the target multi-source remote sensing image pair, calculates the correlation between the nodes in the plurality of nodes to obtain a first correlation matrix of a local hypergraph and a second correlation matrix of a global hypergraph respectively, and forms a plurality of hyperedges of the corresponding hypergraph by using each correlation matrix to construct the local hypergraph and the global hypergraph with the plurality of nodes and the plurality of hyperedges; The hypergraph learning module is used to perform hypergraph learning by using a preset hypergraph neural network based on the first correlation matrix and the second correlation matrix to extract features of the multi-source remote sensing image pair before and after the change, and generate a first probability prediction result of node change of the local hypergraph and a second probability prediction result of node change of the global hypergraph respectively; The fusion prediction module is used to fuse the first probability prediction result and the second probability prediction result to obtain a final fusion prediction result of each pixel, and obtain a final change detection result of the target multi-source remote sensing image pair according to the final fusion prediction result of each pixel.

6. The apparatus of claim 5, wherein, The hypergraph construction module comprises: The segmentation unit is used to segment the plurality of source remote sensing image pairs to generate a plurality of image blocks, and construct the local hypergraph by taking the pixels inside each image block of the plurality of image blocks as nodes, and construct the global hypergraph by taking each image block of the plurality of image blocks as nodes.

7. The apparatus of claim 5, wherein, The hypergraph construction module comprises: The local hypergraph construction unit is used to obtain the first correlation matrix of the local hypergraph, and the calculation formula of the first correlation matrix is: wherein, is the Euclidean distance between the current node and the jth node, is the average Euclidean distance between the current node and all other nodes except the current node, and is the average Euclidean distance between the current node and all other nodes. The calculation unit calculates the spatial distance between the current node and a plurality of spatially adjacent position nodes around the current node by taking the current node as a center point based on the spatial position of each image block in the unsegmented image, and the formula of the spatial distance calculation method is: wherein, and are two-dimensional spatial coordinates of the current node, and are two-dimensional spatial coordinates of a plurality of spatially adjacent location nodes surrounding the current node. The global hypergraph construction unit is used to obtain the second correlation matrix of the global hypergraph by using the spatial distance, and the calculation formula of the second correlation matrix is: in, For the current node and the node Spatial distance, The average spatial distance between the current node and all other neighboring nodes.

8. An electronic device, comprising: Comprise: The memory, the processor and the computer program stored on the memory and executable on the processor, the processor executes the program to realize the hypergraph neural network based multi-source remote sensing image change detection method in any one of claims 1-4.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the hypergraph neural network based multi-source remote sensing image change detection method in any one of claims 1-4.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed for implementing the hypergraph neural network-based multi-source remote sensing image change detection method according to any one of claims 1-4.

Images