Multi-source remote sensing image change detection method and device based on hypergraph neural network

Abstract

The invention relates to the technical field of image processing, in particular to a multi-source remote sensing image change detection method and device based on a hypergraph neural network, and the method comprises the steps: constructing nodes based on a given multi-source remote sensing image pair, calculating the incidence relation between the nodes, obtaining an incidence matrix to form a hyperedge, and obtaining a hyperedge; therefore, a local hypergraph and a global hypergraph with nodes and hyperedges are constructed. And extracting features before and after image change by combining a hypergraph neural network with incidence matrix learning, and outputting probability prediction results of local and global hypergraph node change. And fusing probability prediction results of local and global hypergraph node changes by using a fusion strategy to obtain a final fusion prediction result of each pixel, and finally judging whether each pixel changes or not through a threshold classification method to obtain a final change detection result. Therefore, the multi-source heterogeneous image change detection precision is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for detecting changes in multi-source remote sensing images based on a hypergraph neural network. Background Art

[0002] Among the related technologies, remote sensing images play an important role in surface environment monitoring, disaster emergency management, and urban and rural development planning. Multi-source heterogeneous images can realize real-time and effective change monitoring, but compared with single-modal dual-temporal remote sensing data, their change detection accuracy is lower.

[0003] At present, the change detection of multi-source heterogeneous remote sensing images is mainly divided into methods based on homogeneous transformation and deep feature learning. The method based on homogeneous transformation has poor generalization ability. When the scene is complex or the noise is large, the artificially constructed conversion model between modalities may fail, which leads to a decrease in change detection performance. The method based on deep feature learning requires more data sources for learning, but multi-source heterogeneous remote sensing images have less data of different types, which is difficult to meet the training requirements of deep neural networks and needs to be improved urgently. Summary of the invention

[0004] The present application provides a multi-source remote sensing image change detection method and device based on a hypergraph neural network to solve the problem of low accuracy in multi-source heterogeneous image change detection in related technologies.

[0005] The first aspect of the present application provides a multi-source remote sensing image change detection method based on a hypergraph neural network, comprising the following steps: based on a target multi-source remote sensing image pair, constructing multiple nodes of the target multi-source remote sensing image pair, and calculating the association relationship between the nodes in the multiple nodes to obtain a first association matrix of a local hypergraph and a second association matrix of a global hypergraph, respectively, and using each association matrix to form multiple hyperedges of a corresponding hypergraph to construct a local hypergraph and a global hypergraph having the multiple nodes and the multiple hyperedges; based on the first association matrix and the second association matrix, using a preset hypergraph neural network to perform hypergraph learning to extract features of the multi-source remote sensing image pair before and after the change, and respectively generate a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph; fusing the first probability prediction result and the second probability prediction result to obtain a final fusion prediction result of each pixel, and obtaining a final change detection result of the target multi-source remote sensing image pair according to the final fusion prediction result of each pixel.

[0006] Through the above technical scheme, the embodiment of the present application can construct a local hypergraph and a global hypergraph, use nodes and their association relationships to effectively characterize the characteristics of multi-source remote sensing images before and after changes, and accurately predict the probability of node changes through the learning of hypergraph neural networks, and combine fusion strategies to integrate local and global information. Finally, the threshold classification method is used to determine the changes of each pixel, making the detection results clearer.

[0007] Optionally, in one embodiment of the present application, the constructing of a local hypergraph and a global hypergraph having the multiple nodes and the multiple hyperedges includes: segmenting the multi-source remote sensing image pairs to generate multiple image blocks, constructing the local hypergraph using pixels inside each of the multiple image blocks as nodes, and constructing the global hypergraph using each of the multiple image blocks as a node.

[0008] Through the above technical solution, the embodiment of the present application can effectively capture local features and global information in the image by dividing the multi-source remote sensing image into multiple image blocks, using the pixels inside the image block as nodes to construct a local hypergraph, and using the entire image block as a node to construct a global hypergraph. This hierarchical hypergraph structure not only improves the accuracy of change detection, but also enhances the sensitivity to feature differences between images of different modalities, thereby more accurately identifying change areas in complex scenes and improving the overall performance of remote sensing image change detection.

[0009] Optionally, in one embodiment of the present application, based on the target multi-source remote sensing image pair, constructing multiple nodes of the target multi-source remote sensing image pair, and calculating the association relationship between the nodes in the multiple nodes to obtain a first association matrix of a local hypergraph and a second association matrix of a global hypergraph, respectively, including:

[0010] The calculation formula of the first incidence matrix is:

[0011]

[0012] Among them, dis_Eu i is the Euclidean distance between the current node and the i-th node, dis_avg_E is the average Euclidean distance between the current node and all other nodes except the current node;

[0013] Based on the spatial position of each image block in the unsegmented image, the current node is taken as the center point, and the spatial distance between the current node and multiple spatially adjacent nodes around the current node is calculated using a spatial position calculation method. The calculation formula of the spatial distance is:

[0014]

[0015] Among them, x i andi is the two-dimensional space coordinate of the current node, x j and j The two-dimensional spatial coordinates of multiple spatially adjacent nodes around the current node;

[0016] The second association matrix is ​​obtained by using the spatial distance. The calculation formula of the second association matrix is:

[0017]

[0018] Among them, dis_Su i is the spatial distance between the current node and node i, and dis_avg_s is the average spatial distance between the current node and the remaining neighboring nodes except the current node.

[0019] Through the above technical solution, the embodiment of the present application can construct the association matrix of the local and global hypergraphs, and use the Euclidean distance and spatial distance to effectively capture the association relationship between nodes. The association matrix of the local hypergraph can accurately reflect the structural characteristics inside the image block by calculating the Euclidean distance between the target node and other nodes; while the association matrix of the global hypergraph takes into account the positional relationship of the image block in the overall image through spatial position calculation, thereby enhancing the understanding of spatial features. The advantage of this method is that it can make full use of the characteristics of multi-source heterogeneous remote sensing images, improve the accuracy and robustness of change detection, and thus achieve more accurate change detection results in complex scenes.

[0020] Optionally, in one embodiment of the present application, the hypergraph neural network includes a feature extraction module, an aggregation network module and a classification module, wherein, based on the first association matrix and the second association matrix, a preset hypergraph neural network is used to perform hypergraph learning to extract the features of the multi-source remote sensing image before and after the change, and respectively generate a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph, including: inputting the multiple image blocks and the first association matrix into the preset hypergraph neural network to perform the local hypergraph learning, and respectively extracting the local hypergraph features of the multiple image blocks before and after the change through the feature extraction module, and The local hypergraph features are processed by the aggregation network module to obtain processed local hypergraph features, and the processed local hypergraph features are classified by the classification module to obtain the first probability prediction result; the node data composed of the central pixels of all the multiple image blocks and the second association matrix are input into the preset hypergraph neural network to perform the global hypergraph learning, and the global hypergraph features of the multi-source remote sensing image before and after the change are respectively extracted by the feature extraction module, and the global hypergraph features are processed by the aggregation network module to obtain all processed hypergraph features, and the processed global hypergraph features are classified by the classification module to obtain the second probability prediction result.

[0021] Through the above technical solution, the embodiment of the present application can effectively extract and process the local and global hypergraph features of multi-source remote sensing images through the collaborative work of the feature extraction module, aggregation network module and classification module in the hypergraph neural network, and then accurately predict the probability of node changes. This structure not only improves the ability to capture image features before and after the change, but also enhances the accuracy of change detection by aggregating information at different levels, significantly improving the accuracy of change detection in multi-source heterogeneous remote sensing images.

[0022] The second aspect of the present application provides a multi-source remote sensing image change detection device based on a hypergraph neural network, including: a hypergraph construction module, which is used to construct multiple nodes of the target multi-source remote sensing image pair based on the target multi-source remote sensing image pair, and calculate the association relationship between the nodes in the multiple nodes to obtain a first association matrix of the local hypergraph and a second association matrix of the global hypergraph respectively, and use each association matrix to form multiple hyperedges of the corresponding hypergraph to construct a local hypergraph and a global hypergraph with the multiple nodes and the multiple hyperedges; a hypergraph learning module, which is used to perform hypergraph learning based on the first association matrix and the second association matrix using a preset hypergraph neural network to extract the features of the multi-source remote sensing image pair before and after the change, and respectively generate a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph; a fusion prediction module, which is used to fuse the first probability prediction result and the second probability prediction result to obtain a final fusion prediction result of each pixel, and obtain a final change detection result of the target multi-source remote sensing image pair according to the final fusion prediction result of each pixel.

[0023] Through the above technical scheme, the embodiment of the present application can construct a local hypergraph and a global hypergraph, use nodes and their association relationships to effectively characterize the characteristics of multi-source remote sensing images before and after changes, and accurately predict the probability of node changes through the learning of hypergraph neural networks, and combine fusion strategies to integrate local and global information. Finally, the threshold classification method is used to determine the changes of each pixel, making the detection results clearer.

[0024] Optionally, in one embodiment of the present application, the hypergraph construction module includes: a segmentation unit, used to segment the multi-source remote sensing image pair to generate multiple image blocks, using pixels inside each image block of the multiple image blocks as nodes to construct the local hypergraph, and using each image block of the multiple image blocks as a node to construct the global hypergraph.

[0025] Through the above technical solution, the embodiment of the present application can effectively capture local features and global information in the image by dividing the multi-source remote sensing image into multiple image blocks, using the pixels inside the image block as nodes to construct a local hypergraph, and using the entire image block as a node to construct a global hypergraph. This hierarchical hypergraph structure not only improves the accuracy of change detection, but also enhances the sensitivity to feature differences between images of different modalities, thereby more accurately identifying change areas in complex scenes and improving the overall performance of remote sensing image change detection.

[0026] Optionally, in one embodiment of the present application, the hypergraph construction module includes: a local hypergraph construction unit, used to obtain a first association matrix of the local hypergraph, and the calculation formula of the first association matrix is:

[0027]

[0028] Among them, dis_Eu i is the Euclidean distance between the current node and the i-th node, dis_avg_E is the average Euclidean distance between the current node and all other nodes except the current node;

[0029] A calculation unit is used to calculate the spatial distance between the current node and a plurality of spatially adjacent nodes around the current node based on the spatial position of each image block in the unsegmented image and taking the current node as the center point by using a spatial position calculation method, wherein the calculation formula of the spatial distance is:

[0030]

[0031] Among them, x i and i is the two-dimensional space coordinate of the current node, x j and j The two-dimensional spatial coordinates of multiple spatially adjacent nodes around the current node;

[0032] A global hypergraph construction unit is used to obtain the second association matrix using the spatial distance, and the calculation formula of the second association matrix is:

[0033]

[0034] Among them, dis_Su i is the spatial distance between the current node and node i, and dis_avg_S is the average spatial distance between the current node and the remaining neighboring nodes except the current node.

[0035] Through the above technical solution, the embodiment of the present application can construct the association matrix of the local and global hypergraphs, and effectively capture the association relationship between nodes using Euclidean distance and spatial distance. The association matrix of the local hypergraph can accurately reflect the structural characteristics inside the image block by calculating the Euclidean distance between the target node and other nodes; while the association matrix of the global hypergraph takes into account the positional relationship of the image block in the overall image through spatial position calculation, thereby enhancing the understanding of spatial features. The advantage of this method is that it can make full use of the characteristics of multi-source heterogeneous remote sensing images, improve the accuracy and robustness of change detection, and thus achieve more accurate change detection results in complex scenes.

[0036] Optionally, in one embodiment of the present application, the hypergraph neural network includes a feature extraction module, an aggregation network module and a classification module, wherein the hypergraph learning module includes: a local hypergraph learning unit, which is used to input the multiple image blocks and the first association matrix into the preset hypergraph neural network to perform the local hypergraph learning, and extract the local hypergraph features of the multiple image blocks before and after the change respectively through the feature extraction module, and process the local hypergraph features through the aggregation network module to obtain the processed local hypergraph features, and use the classification module to classify the processed local hypergraph features to obtain the first probability prediction result; a global hypergraph learning unit, which is used to input the node data composed of the central pixels of all the multiple image blocks and the second association matrix into the preset hypergraph neural network to perform the global hypergraph learning, and extract the global hypergraph features of the multi-source remote sensing image before and after the change respectively through the feature extraction module, and process the global hypergraph features through the aggregation network module to obtain all processed hypergraph features, and use the classification module to classify the processed global hypergraph features to obtain the second probability prediction result.

[0037] Through the above technical solution, the embodiment of the present application can effectively extract and process the local and global hypergraph features of multi-source remote sensing images through the collaborative work of the feature extraction module, aggregation network module and classification module in the hypergraph neural network, and then accurately predict the probability of node changes. This structure not only improves the ability to capture image features before and after the change, but also enhances the accuracy of change detection by aggregating information at different levels, significantly improving the accuracy of change detection in multi-source heterogeneous remote sensing images.

[0038] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-source remote sensing image change detection method based on a hypergraph neural network as described in the above embodiment.

[0039] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned multi-source remote sensing image change detection method based on a hypergraph neural network.

[0040] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned multi-source remote sensing image change detection method based on a hypergraph neural network.

[0041] The embodiments of the present application can construct local and global hypergraphs, and use nodes and their associations to effectively extract the features of multi-source remote sensing images before and after changes, thereby achieving high-precision change detection. The local hypergraph is constructed through the internal pixels of the image block to accurately reflect the structural characteristics of the image block; the global hypergraph uses the entire image block as a node, considers the spatial position relationship, and enhances the understanding of spatial features. The feature extraction, aggregation, and classification modules of the hypergraph neural network work together to improve the ability to capture image features before and after changes, and by fusing local and global information, improve the reliability and accuracy of change detection. Finally, the threshold classification method is used to judge the changes in each pixel, making the detection results clearer. This method makes full use of the characteristics of multi-source heterogeneous remote sensing images to achieve more accurate change detection in complex scenes and significantly improve detection accuracy.

[0042] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0044] Figure 1 A flowchart of a multi-source remote sensing image change detection method based on a hypergraph neural network provided according to an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a hypergraph neural network structure according to an embodiment of the present application;

[0046] Figure 3 A public dataset used for implementing change detection according to a specific embodiment of the present application;

[0047] Figure 4 A schematic diagram showing a comparison between a change detection result according to a specific embodiment of the present application and the results of other detection methods;

[0048] Figure 5 A schematic diagram of the structure of a multi-source remote sensing image change detection device based on a hypergraph neural network provided according to an embodiment of the present application;

[0049] Figure 6 The figure is a structural example diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0051] The following describes the multi-source remote sensing image change detection method and device based on a hypergraph neural network according to an embodiment of the present application with reference to the accompanying drawings. In view of the problem of low detection accuracy in multi-source heterogeneous image change detection in the related technologies mentioned in the above background technology, the present application provides a multi-source remote sensing image change detection method based on a hypergraph neural network. In this method, local hypergraphs and global hypergraphs can be constructed, and nodes and their associations can be used to effectively characterize the features of multi-source remote sensing images before and after the change. The probability of node change can be accurately predicted through the learning of the hypergraph neural network, and the fusion strategy is combined to integrate local and global information. Finally, the threshold classification method is used to judge the change of each pixel, so that the detection result is clearer. In this way, the problem of low accuracy in multi-source heterogeneous image change detection in the related technologies is solved.

[0052] Specifically, Figure 1 A flow chart of a multi-source remote sensing image change detection method based on a hypergraph neural network provided in an embodiment of the present application.

[0053] like Figure 1 As shown, the multi-source remote sensing image change detection method based on hypergraph neural network includes the following steps:

[0054] In step S101, based on the target multi-source remote sensing image pair, multiple nodes of the target multi-source remote sensing image pair are constructed, and the correlation relationship between the nodes in the multiple nodes is calculated to obtain a first correlation matrix of the local hypergraph and a second correlation matrix of the global hypergraph, respectively, and each correlation matrix is ​​used to form multiple hyperedges of the corresponding hypergraph to construct a local hypergraph and a global hypergraph with multiple nodes and multiple hyperedges.

[0055] It can be understood that the process of hypergraph construction is actually to convert multi-source remote sensing images into a hypergraph structure with nodes and hyperedges. The nodes are usually pixels in the image, and the hyperedges represent the association relationship between image pixels.

[0056] Optionally, in one embodiment of the present application, a local hypergraph and a global hypergraph having multiple nodes and multiple hyperedges are constructed, including: segmenting multi-source remote sensing image pairs to generate multiple image blocks, constructing a local hypergraph using pixels inside each image block of the multiple image blocks as nodes, and constructing a global hypergraph using each image block of the multiple image blocks as a node.

[0057] In the actual implementation process, the image segmentation method can solve the problem of large size of remote sensing images for detecting changes. The image is divided into p×p image blocks, and then the pixels and image blocks inside the image blocks are used as nodes to hierarchically construct local and global hypergraphs.

[0058] On the one hand, the local hypergraph is constructed by taking the pixels inside an image block as nodes, and the association relationship of the hyperedge is measured using the Euclidean distance. Specifically, for the target node, the Euclidean distance between all other nodes in the image block and the node is calculated, and then the nearest K nodes are taken as the nodes associated with the target node on a hyperedge. To obtain the association matrix of the local hypergraph, the following formula is used to calculate the association weight:

[0059]

[0060] Among them, dis_Eu i is the Euclidean distance between the target node and the i-th node, and dis_avg_E is the average Euclidean distance between the target node and all other nodes in the image block. Subsequently, the weights of the K nodes with the closest Euclidean distance to the target node are stored in the form of a sparse matrix, where the value of the main diagonal element is 1, indicating that each node has the greatest correlation with itself, generating the correlation matrix H of the local hypergraph local .

[0061] On the other hand, when constructing the global hypergraph, the image block as a whole is taken as a node. In particular, the embodiment of the present application can assume that under the condition of a certain size p (p<15), the difference between the central pixel and the surrounding neighborhood pixels of the image block is small and can represent the entire image block. Therefore, the central pixel is used as the node of the global hypergraph, and the probability value of the final predicted node change is assigned to the entire image block. The hyperedge association relationship of the global hypergraph is mainly based on the spatial position relationship of each node. Specifically, the spatial position of each image block in the unsegmented image is maintained, and then the target node position is used as the center point to calculate the distance between it and the nodes at the surrounding 8 spatially adjacent positions. For the target node j, the distance between node i and it is calculated according to the spatial position, and the formula is as follows:

[0062]

[0063] Among them, x i and i is the two-dimensional space coordinate of the target node, x j and j is the two-dimensional spatial coordinate of multiple neighborhood nodes in space. Furthermore, the spatial distance is used to obtain the association weight of the global hypergraph, and the calculation formula is:

[0064]

[0065] Among them, dis_Su i is the spatial distance between the target node and node i, and dis_avg_S is the average spatial distance between the target node and the remaining neighboring nodes. Similarly, the weights between the eight spatially adjacent nodes are stored in the form of a sparse matrix, and the values ​​of the main diagonal elements are still 1, generating the global hypergraph association matrix H global .

[0066] The embodiments of the present application can form the hyperedge structure of the local hypergraph and the global hypergraph by constructing nodes and calculating the association relationship between nodes for a given multi-source remote sensing image pair, thereby effectively converting the image data into a hypergraph form. Specifically, the local hypergraph uses the pixels inside the image block as nodes, and uses the Euclidean distance to measure the association between nodes to generate the association matrix of the local hypergraph; the global hypergraph uses the entire image block as a node, uses the center pixel to represent the entire image block, calculates the association weight based on the spatial position relationship, and generates the association matrix of the global hypergraph. This hierarchical hypergraph construction method can not only handle the complexity of large-scale remote sensing images, but also improve the accuracy of change detection.

[0067] In step S102, based on the first association matrix and the second association matrix, a preset hypergraph neural network is used to perform hypergraph learning to extract features of the multi-source remote sensing image pairs before and after the change, and to generate a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph, respectively.

[0068] In the embodiment of the present application, the update formula of the hypergraph neural network is as follows:

[0069]

[0070] Where H represents the hypergraph association matrix; D e and D v They represent the diagonal matrices composed of the hyperedges and node degrees of the hypergraph respectively; W is the weight matrix of the hypergraph, which is initialized to the unit matrix; θ represents the network parameters of the hypergraph neural network; X is the node data or hypergraph feature input to the hypergraph convolution layer; Y is the hypergraph feature obtained by X through a layer of hypergraph convolution layer.

[0071] Optionally, in one embodiment of the present application, the hypergraph neural network includes a feature extraction module, an aggregation network module and a classification module, wherein, based on the first association matrix and the second association matrix, a preset hypergraph neural network is used to perform hypergraph learning to extract features of multi-source remote sensing images before and after the change, and respectively generate a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph, including: inputting multiple image blocks and the first association matrix into the preset hypergraph neural network for local hypergraph learning, and extracting the local hypergraph features of the multiple image blocks before and after the change through the feature extraction module. Features, and process the local hypergraph features through the aggregation network module to obtain processed local hypergraph features, and use the classification module to classify the processed local hypergraph features to obtain a first probability prediction result; the node data composed of the central pixels of all multiple image blocks and the second association matrix are input into the preset hypergraph neural network to perform global hypergraph learning, and the global hypergraph features before and after the change of the multi-source remote sensing image are extracted respectively through the feature extraction module, and the global hypergraph features are processed through the aggregation network module to obtain all processed hypergraph features, and the classification module is used to classify the processed global hypergraph features to obtain a second probability prediction result.

[0072] Specifically, this application constructs a network structure in which two hypergraph neural networks extract the hypergraph features of dual-temporal remote sensing images respectively and then perform aggregation processing, such as Figure 2 As shown. First, for local hypergraph learning, the image block node data group of the dual-phase image segmentation and the hypergraph association matrix group constructed by each image block are respectively input into the two-way hypergraph neural network. After extracting features through several layers of hypergraph convolution and nonlinear activation layers, the hypergraph features of the image before and after the change are obtained. Then the extracted hypergraph features are processed by the aggregation network module, and finally the classification module is used to predict the probability of node change. The loss function of local hypergraph learning is focal loss, and its formula is as follows:

[0073] Focal_loss(p t )=-α t (1-p t ) γ log(p t )(5)

[0074] Among them, p t is the probability of the positive class, which in this article refers to the probability of change, α t is the category balancing weight, and γ is a hyperparameter for adjusting the weight of simple samples to be reduced.

[0075] The global hypergraph learning uses the same network. The inputs to the two networks are the node data composed of the central pixels of the image blocks of the dual-temporal remote sensing images and the obtained global hypergraph association matrix. The network loss function used for training global hypergraph learning is the binary cross entropy loss, and the formula is as follows:

[0076]

[0077] Where N represents the number of nodes, y i Indicates the category to which the i-th sample belongs, p i Represents the predicted probability value of the i-th sample.

[0078] The embodiment of the present application can effectively extract the features corresponding to the multi-source remote sensing images before and after the change by using a hypergraph neural network and combining it with a hypergraph association matrix for hypergraph learning, and output the probability prediction results of the local and global hypergraph node changes. This method constructs a network structure composed of a two-way hypergraph neural network, which performs feature extraction, aggregation and classification on local and global hypergraphs respectively, thereby improving the accuracy of change detection. By using focal loss and binary cross entropy loss as loss functions, the robustness of the model in processing unbalanced samples is further enhanced. This method not only improves the accuracy of change detection, but also can effectively cope with the complexity of multi-source heterogeneous remote sensing images.

[0079] In step S103, the first probability prediction result and the second probability prediction result are fused to obtain a final fused prediction result for each pixel, and a final change detection result of the target multi-source remote sensing image pair is obtained according to the final fused prediction result for each pixel.

[0080] Specifically, through the training and learning of the hypergraph neural network, local and global hypergraph node prediction results are obtained. The local nodes are the pixels of the image block, and the prediction results can be directly spliced; the global hypergraph predicts the results of the central pixels of the image block, which are assigned to the entire image block and then all the image blocks are spliced. Finally, the local and global change prediction results are fused using a fusion method. After fusion, the threshold classification method is used to determine whether each pixel has changed to obtain the final change detection result.

[0081] The embodiment of the present application can make full use of information at different levels and improve the accuracy of change detection by fusing the change probability prediction results of local and global hypergraph nodes. The local hypergraph provides detailed pixel-level change information, while the global hypergraph takes into account the overall characteristics of the image block. The combination of the two makes the detection results more comprehensive and accurate. In addition, the threshold classification method is used for final judgment, which simplifies the result processing flow and ensures the efficiency and practicality of change detection.

[0082] The present application is described in detail below with reference to a specific embodiment.

[0083] 1) Input two dual-temporal heterogeneous remote sensing images, which are images of the same area taken at different times using different sensors, and there are actual changes between the two images. Figure 3 The dataset shown is a flood change detection dataset for the Gloucester area in the UK. Synthetic aperture radar images were collected before the change, and normalized vegetation index images were collected after the change. This image pair is a multi-source heterogeneous remote sensing image pair with changes in the same area at different times.

[0084] 2) The image pair is segmented. In this embodiment, the segmentation size is 7×7, so there are 49 pixels in each image block. If the size of the original image cannot be divided evenly, zero padding is performed on the right and bottom of the image.

[0085] 3) Construction of local hypergraph: The pixels in each image block are regarded as hypergraph nodes, and the Euclidean distance between each node and the remaining nodes is calculated. Then, according to formula (1), the nearest K nodes are calculated as nodes on a hyperedge. In this embodiment, K is 10 to obtain the local hypergraph association matrix of each image block.

[0086] 4) Construction of the global hypergraph: The central pixel of each image block is taken as a node, and the spatial distance between the target node and the surrounding 8 neighborhoods is calculated according to formula (2) to obtain the global hypergraph association matrix of the entire image.

[0087] 5) Select a certain proportion of positive and negative sample image blocks as training sets. In this embodiment, 100 positive and negative samples are selected. The hypergraph data nodes and hypergraph association matrix constructed by the dual-phase image are input into the following Figure 2 In the hypergraph neural network constructed as shown, after training and learning, the node change probability values ​​predicted by local and global hypergraph learning are obtained respectively.

[0088] 6) Use a fusion strategy or method to fuse the local and global acquired probability values ​​of change. In this embodiment, the fusion strategy adopts the DS (Dempster-Shafer, Dempster synthesis rule) evidence theory, in which the hypergraph neural network's prediction results for local and global are used as the trust function of the pixel change and unchanged. Assume that A represents a change, m1 and m2 represent two quality functions of the local and global hypergraphs, respectively, and represent the probability values ​​of the change predicted by the local and global hypergraph learning, respectively. According to the DS synthesis rule, the probability of change after fusion is:

[0089]

[0090] 7) According to the fused predicted probability value, the threshold segmentation method is used to obtain the final change detection binary result map. Figure 4 As shown, the first picture is the true value, the second picture is the detection result of the HPT (Homogeneous Transformation) method, the third picture is the detection result of the SCCN (Spatial Contextual Convolutional Network) method, and the fourth picture is the detection result of the method of the present invention, among which HPT is a representative method based on homogeneous transformation, and SCCN is a representative method based on deep feature learning.

[0091] Based on the experimental results, it is shown that the multi-source remote sensing image change detection method based on hypergraph neural network proposed in this application can obtain good detection results and can effectively improve the detection accuracy compared with other methods.

[0092] To further demonstrate the effect of the present invention, Table 1 shows the numerical comparison results of the method used in the present invention and other methods. Among them, the calculation formulas and Table 1 of each numerical evaluation index are as follows:

[0093]

[0094] Table 1

[0095] method OA F1 KC HPT 96.15 82.48 80.34 SCCN 95.22 77.65 75.02 This application method 98.63 94.34 93.57

[0096] It can be seen from Table 1 that, consistent with the conclusion of the visual results, the multi-source remote sensing image change detection method based on hypergraph neural network proposed in this application obtains the optimal detection index performance and can effectively improve the accuracy of change detection of multi-source heterogeneous remote sensing images.

[0097] According to the multi-source remote sensing image change detection method based on hypergraph neural network proposed in the embodiment of the present application, it is possible to construct local hypergraphs and global hypergraphs, and use nodes and their associations to effectively characterize the characteristics of multi-source remote sensing images before and after the change. The probability of node change can be accurately predicted through the learning of the hypergraph neural network, and the fusion strategy is combined to integrate local and global information. Finally, the threshold classification method is used to determine the change of each pixel, making the detection result clearer.

[0098] Next, a multi-source remote sensing image change detection device based on a hypergraph neural network proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0099] Figure 5 It is a block diagram of a multi-source remote sensing image change detection device based on a hypergraph neural network according to an embodiment of the present application.

[0100] like Figure 5As shown, the multi-source remote sensing image change detection device 10 based on the hypergraph neural network includes: a hypergraph construction module 100, a hypergraph learning module 200 and a fusion prediction module 300.

[0101] Specifically, the hypergraph construction module 100 is used to construct multiple nodes of the target multi-source remote sensing image pair based on the target multi-source remote sensing image pair, and calculate the association relationship between the nodes in the multiple nodes to obtain a first association matrix of the local hypergraph and a second association matrix of the global hypergraph, respectively, and use each association matrix to form multiple hyperedges of the corresponding hypergraph to construct a local hypergraph and a global hypergraph with multiple nodes and multiple hyperedges.

[0102] The hypergraph learning module 200 is used to perform hypergraph learning based on the first association matrix and the second association matrix using a preset hypergraph neural network to extract the features of the multi-source remote sensing image pairs before and after the change, and generate a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph, respectively.

[0103] The fusion prediction module 300 is used to fuse the first probability prediction result and the second probability prediction result to obtain the final fusion prediction result of each pixel, and obtain the final change detection result of the target multi-source remote sensing image pair according to the final fusion prediction result of each pixel.

[0104] Optionally, in one embodiment of the present application, the hypergraph construction module 100 includes: a segmentation unit, used to segment and divide multi-source remote sensing image pairs to generate multiple image blocks, construct a local hypergraph with pixels inside each image block of the multiple image blocks as nodes, and construct a global hypergraph with each image block of the multiple image blocks as a node.

[0105] Optionally, in one embodiment of the present application, the hypergraph construction module 100 includes: a local hypergraph construction unit, a computing unit and a global hypergraph construction unit.

[0106] The local hypergraph construction unit is used to obtain a first association matrix of the local hypergraph. The calculation formula of the first association matrix is:

[0107]

[0108] Among them, dis_Eu i is the Euclidean distance between the current node and the i-th node, and dis_avg_E is the average Euclidean distance between the current node and all other nodes except the current node.

[0109] The calculation unit is used to calculate the spatial distance between the current node and multiple spatially adjacent nodes around the current node based on the spatial position of each image block in the unsegmented image and the current node as the center point using a spatial position calculation method. The calculation formula of the spatial distance is:

[0110]

[0111] Among them, x i and i is the two-dimensional space coordinate of the current node, x j and j It is the two-dimensional spatial coordinates of multiple spatially adjacent nodes around the current node.

[0112] The global hypergraph construction unit is used to obtain the second association matrix using the spatial distance. The calculation formula of the second association matrix is:

[0113]

[0114] Among them, dis_Su i is the spatial distance between the current node and node i, and dis_avg_S is the average spatial distance between the current node and the remaining neighboring nodes except the current node.

[0115] Optionally, in one embodiment of the present application, the hypergraph neural network includes a feature extraction module, an aggregation network module and a classification module, wherein the hypergraph learning module 200 includes: a local hypergraph learning unit and a global hypergraph learning unit.

[0116] Among them, the local hypergraph learning unit is used to input multiple image blocks and the first association matrix into a preset hypergraph neural network to perform local hypergraph learning, and extract the local hypergraph features of the multiple image blocks before and after the change through the feature extraction module, and process the local hypergraph features through the aggregation network module to obtain the processed local hypergraph features, and use the classification module to classify the processed local hypergraph features to obtain the first probability prediction result.

[0117] A global hypergraph learning unit is used to input the node data composed of the central pixels of all multiple image blocks and the second association matrix into a preset hypergraph neural network to perform global hypergraph learning, and extract the global hypergraph features of the multi-source remote sensing image before and after the change through the feature extraction module, and process the global hypergraph features through the aggregation network module to obtain all the processed hypergraph features, and use the classification module to classify the processed global hypergraph features to obtain a second probability prediction result.

[0118] It should be noted that the aforementioned explanation of the embodiment of the multi-source remote sensing image change detection method based on a hypergraph neural network is also applicable to the multi-source remote sensing image change detection device based on a hypergraph neural network in this embodiment, and will not be repeated here.

[0119] According to the multi-source remote sensing image change detection device based on a hypergraph neural network proposed in the embodiment of the present application, it is possible to construct a local hypergraph and a global hypergraph, and use nodes and their associations to effectively characterize the features of the multi-source remote sensing images before and after the change. The probability of node change can be accurately predicted through the learning of the hypergraph neural network, and the local and global information can be integrated in combination with the fusion strategy. Finally, the threshold classification method is used to determine the change of each pixel, making the detection result clearer.

[0120] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0121] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0122] When the processor 602 executes the program, the multi-source remote sensing image change detection method based on the hypergraph neural network provided in the above embodiment is implemented.

[0123] Furthermore, the electronic device further comprises:

[0124] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0125] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0126] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0127] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0128] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0129] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0130] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-source remote sensing image change detection method based on a hypergraph neural network as described above.

[0131] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned multi-source remote sensing image change detection method based on a hypergraph neural network.

[0132] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0133] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0134] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0136] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0137] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0139] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-source remote sensing image change detection method based on hypergraph neural network, characterized in that: The following steps are involved: Based on a target multi-source remote sensing image pair, a plurality of nodes of the target multi-source remote sensing image pair are constructed, and association relationships between the nodes in the plurality of nodes are calculated to obtain a first association matrix of a local hypergraph and a second association matrix of a global hypergraph, respectively, and a plurality of hyperedges of a corresponding hypergraph are formed using each association matrix to construct a local hypergraph and a global hypergraph having the plurality of nodes and the plurality of hyperedges; Based on the first association matrix and the second association matrix, a preset hypergraph neural network is used to perform hypergraph learning to extract features of the multi-source remote sensing image pair before and after the change, and a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph are generated respectively; The first probability prediction result and the second probability prediction result are fused to obtain a final fused prediction result for each pixel, and a final change detection result of the target multi-source remote sensing image pair is obtained based on the final fused prediction result for each pixel.

2. The method according to claim 1, characterized in that The constructing of a local hypergraph and a global hypergraph having the plurality of nodes and the plurality of hyperedges comprises: The multi-source remote sensing image pair is segmented to generate a plurality of image blocks, the local hypergraph is constructed with pixels inside each of the plurality of image blocks as nodes, and the global hypergraph is constructed with each of the plurality of image blocks as nodes.

3. The method according to claim 2, characterized in that The method of constructing a plurality of nodes of the target multi-source remote sensing image pair based on the target multi-source remote sensing image pair, and calculating the association relationship between the nodes in the plurality of nodes to obtain a first association matrix of a local hypergraph and a second association matrix of a global hypergraph respectively includes: The calculation formula of the first incidence matrix is: Among them, dis_Eu i is the Euclidean distance between the current node and the i-th node, dis_avg_E is the average Euclidean distance between the current node and all other nodes except the current node; Based on the spatial position of each image block in the unsegmented image, the current node is taken as the center point, and the spatial distance between the current node and multiple spatially adjacent nodes around the current node is calculated using a spatial position calculation method. The calculation formula of the spatial distance is: Among them, x i and i is the two-dimensional space coordinate of the current node, x j and j The two-dimensional spatial coordinates of multiple spatially adjacent nodes around the current node; The second association matrix is ​​obtained by using the spatial distance. The calculation formula of the second association matrix is: Among them, dis_Su i is the spatial distance between the current node and node i, and dis_avg_S is the average spatial distance between the current node and the remaining neighboring nodes except the current node.

4. The method according to claim 2, characterized in that: The hypergraph neural network comprises a feature extraction module, an aggregation network module and a classification module, wherein the hypergraph learning is performed using a preset hypergraph neural network based on the first association matrix and the second association matrix to extract the features of the multi-source remote sensing image pair before and after the change, and respectively generate a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph, including: Inputting the multiple image blocks and the first association matrix into the preset hypergraph neural network to perform the local hypergraph learning, and respectively extracting the local hypergraph features of the multiple image blocks before and after the change through the feature extraction module, and processing the local hypergraph features through the aggregation network module to obtain processed local hypergraph features, and using the classification module to classify the processed local hypergraph features to obtain the first probability prediction result; The node data composed of the central pixels of all the multiple image blocks and the second association matrix are input into the preset hypergraph neural network to perform the global hypergraph learning, and the global hypergraph features of the multi-source remote sensing image before and after the change are extracted respectively by the feature extraction module, and the global hypergraph features are processed by the aggregation network module to obtain all the processed hypergraph features, and the processed global hypergraph features are classified by the classification module to obtain the second probability prediction result.

5. A multi-source remote sensing image change detection device based on a hypergraph neural network, characterized in that: include: A hypergraph construction module, based on a target multi-source remote sensing image pair, constructs a plurality of nodes of the target multi-source remote sensing image pair, and calculates association relationships between the nodes in the plurality of nodes, so as to obtain a first association matrix of a local hypergraph and a second association matrix of a global hypergraph, respectively, and uses each association matrix to form a plurality of hyperedges of a corresponding hypergraph, so as to construct a local hypergraph and a global hypergraph having the plurality of nodes and the plurality of hyperedges; A hypergraph learning module, configured to perform hypergraph learning using a preset hypergraph neural network based on the first association matrix and the second association matrix to extract features of the multi-source remote sensing image pair before and after the change, and to generate a first probability prediction result of the node change of the local hypergraph and a second probability prediction result of the node change of the global hypergraph, respectively; A fusion prediction module is used to fuse the first probability prediction result and the second probability prediction result to obtain a final fusion prediction result of each pixel, and obtain a final change detection result of the target multi-source remote sensing image pair based on the final fusion prediction result of each pixel.

6. The device according to claim 5, characterized in that The hypergraph construction module comprises: A segmentation unit is used to segment the multi-source remote sensing image pair to generate a plurality of image blocks, to construct the local hypergraph using pixels inside each of the plurality of image blocks as nodes, and to construct the global hypergraph using each of the plurality of image blocks as a node.

7. The device according to claim 5, characterized in that The hypergraph construction module comprises: The local hypergraph construction unit is used to obtain a first incidence matrix of the local hypergraph, and the calculation formula of the first incidence matrix is: Among them, dis_Eu i is the Euclidean distance between the current node and the i-th node, dis_avg_E is the average Euclidean distance between the current node and all other nodes except the current node; The calculation unit calculates the spatial distance between the current node and a plurality of spatially adjacent nodes around the current node using a spatial position calculation method based on the spatial position of each image block in the unsegmented image and taking the current node as the center point. The spatial distance calculation method formula is: Among them, x i and i is the two-dimensional space coordinate of the current node, x j and j The two-dimensional spatial coordinates of multiple spatially adjacent nodes around the current node; A global hypergraph construction unit is used to obtain a second association matrix of the global hypergraph using the spatial distance, wherein the calculation formula of the second association matrix is: Among them, dis_Su i is the spatial distance between the current node and node i, and dis_avg_S is the average spatial distance between the current node and the remaining neighboring nodes except the current node.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-source remote sensing image change detection method based on a hypergraph neural network as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the multi-source remote sensing image change detection method based on a hypergraph neural network as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the multi-source remote sensing image change detection method based on a hypergraph neural network as described in any one of claims 1 to 4.

Images