Multi-temporal information jointed hyperspectral change detection method based on graph transformer guidance

Abstract

This invention discloses a multi-temporal information joint hyperspectral change detection method based on graph transformer guidance. By constructing a topological graph, the network can propagate labels to unlabeled samples in a semi-supervised learning mode, reducing the dependence on a large number of training samples during training. A graph transformer module with significant relationship enhancement is proposed. This module applies the transformer to a graph structure with superpixels as nodes, enabling the acquisition of affinity relationships between any two sub-regions under low computational cost, breaking through the traditional definition of adjacency relationships and capturing change information in the entire image. A gated change information fusion unit is proposed, which aims to effectively inject the change features guided by the GTrans module into the original dual-temporal stitching features, realizing the organic fusion of multi-temporal information.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing, and in particular to a method for detecting multi-temporal information combined with hyperspectral changes based on graph transformer guidance. Background Technology

[0002] With the rapid development of spectral imaging technology, hyperspectral remote sensing not only acquires spatial information about ground features but also captures continuous spectral information. The "image-spectrum integration" nature of hyperspectral imagery not only provides image data for identifying heterogeneous features with significant spectral differences and homogeneous features with minor differences, but also provides strong evidence for distinguishing between "same feature, different spectra" and "different features, same spectrum." Hyperspectral image change detection effectively monitors dynamic changes and spatial differences in the Earth's surface by analyzing hyperspectral images acquired at the same geographical location but at different times. Due to its applications in various fields such as urban expansion, disaster monitoring, and map production, it has become one of the research hotspots in the field of remote sensing in recent years.

[0003] In recent years, driven by significantly improved computing power and massive amounts of remote sensing data, the rise of deep learning technology has brought new solutions to the intelligent processing of remote sensing images, and change detection methods have begun to explore automation. This characteristic—that the network automatically learns the required features as the basis for detection without relying on manual feature extraction experience—is one of the reasons why deep learning algorithms achieve high detection accuracy. Among them, the local perception and parameter sharing mechanism unique to convolutional neural networks can efficiently abstract concrete image data into high-level features, demonstrating excellent detection performance and unlimited potential. However, these methods are limited by the following factors: First, they mostly use small hyperspectral patches as detection units, but these regular square patches are not universally applicable to complex ground structures and edges, and limiting feature extraction to this fixed patch hinders the enhancement of similar change information from a distance to the current area. Furthermore, the detection accuracy of these methods also depends on a large number of training samples; however, the mixed pixel phenomenon makes hyperspectral image interpretation difficult, and sample labels are not easy to obtain, resulting in detection performance being limited by a finite number of training samples.

[0004] Considering the aforementioned issues, recent studies have incorporated graph theory into this field, aiming to utilize semi-supervised learning models of graphs to propagate labels to nodes connected to the current node that share homogeneous characteristics. By constructing a graph propagation model across the entire hyperspectral image with superpixels as nodes, accurate edge segmentation of ground features can be ensured without sacrificing timeliness. However, research on hyperspectral image processing based on graph neural networks is still in its infancy. How to mine and utilize multi-temporal information in hyperspectral change detection methods based on graph neural networks remains a significant area of ​​research and is one of the key challenges in this field. Summary of the Invention

[0005] The present invention aims to solve the aforementioned technical problems existing in the prior art by providing a multi-temporal information joint hyperspectral change detection method based on graph transformer guidance.

[0006] The technical solution of this invention is: a multi-temporal information joint hyperspectral change detection method based on graph transformer guidance, which is carried out in the following steps:

[0007] Step 1. Denote the hyperspectral image X1∈R at time T1. H×W×C T2 time-phase hyperspectral image X2∈R H×W×C The differential hyperspectral image X3∈R is obtained according to formula (1). H×W×C Where H is the image height, W is the image width, and C is the number of image channels:

[0008] X3 = |X2 - X1| (1);

[0009] Step 2. Preprocess X1, X2, and X3: Convert the dimensions of X1, X2, and X3 to R. HW×C Where HW = H × W, the data of the transformed dimensions are standardized using a pre-packaged data standardization method to make them conform to a normal distribution. The standardized data are denoted as follows: and

[0010] Step 3. Based on the label graph g∈R H×W Construct the label encoding map and label mask map in the training set, validation set and test set;

[0011] Step 3.1. Randomly select 1% of the points in g as training data, and generate a dataset of size R. H×W The training set label image z1 is obtained by using a two-dimensional array z1 consisting entirely of zeros. When the point at position (i, j) is a changing pixel, z1(i, j) is 1; when the point at position (i, j) is a non-changing pixel, z1(i, j) is 2.

[0012] Step 3.2. Then, randomly select the same number of points as the training data from the remaining points as validation data, and generate a dataset of size R. H×W The two-dimensional array z2, consisting entirely of zeros, is set to 1 when the point at position (i, j) is a changing pixel and to 2 when the point at position (i, j) is a constant pixel, thus obtaining the validation set label image z2.

[0013] Step 3.3. The remaining points in g are the test data, generating a dataset of size R. H×WThe test set label image z3 is obtained by using a two-dimensional array z3 consisting entirely of zeros. When the point at position (i, j) is a changing pixel, z3(i, j) is 1; when the point at position (i, j) is a non-changing pixel, z3(i, j) is 2.

[0014] Step 3.4. Perform dummy variable encoding on z1, z2, and z3 sequentially, that is, encode the 0 point at position (i, j) in the graph as 00, the 1 point at position (i, j) in the graph as 10, and the 2 point at position (i, j) in the graph as 01, and further convert their dimensions to R. HW×2 The training set label encoding maps will be obtained respectively. Validation set label encoding diagram and test set label encoding graph

[0015] Step 3.5. Generate a space with size R HW×2 Given a two-dimensional array m1 containing only 1s, determine... If all pixel values ​​in the i-th row are 0, then all pixel values ​​in the i-th row of m1 are changed to 0 to obtain the training set label mask image m1.

[0016] Step 3.6. Generate a space with size R HW×2 Given a two-dimensional array m2 consisting entirely of 1s, determine... If all pixel values ​​in the i-th row are 0, then modify all pixel values ​​in the i-th row of m2 to 0 to obtain the verification set label mask image m2.

[0017] Step 3.7. Generate a space with size R HW×2 Given a two-dimensional array m3 containing only 1s, determine... If all pixel values ​​in the i-th row are 0, then modify all pixel values ​​in the i-th row of m3 to 0 to obtain the test set label mask image m3.

[0018] Step 4. Superpixel segmentation X3

[0019] Step 4.1. Normalize the value of X3 to the interval [-1, 1], apply a Gaussian filter with a window size of 5×5, and then convert its size to R. HW×C The result is denoted as X3′;

[0020] Step 4.2. Call the encapsulated PCA() function to perform principal component analysis on X3′, and take the first three principal components to reduce the dimensionality of X3′. The result is denoted as X3′. a ;

[0021] Step 4.3. Process X3 according to formula (2). a And convert its value to uint8 type, and the size to R. H×W×3The result is denoted as P:

[0022]

[0023] Step 4.4. Perform histogram equalization on each channel of P and enhance its contrast using a filtering function. The result is denoted as X3. b ;

[0024] Step 4.5. Call the encapsulated SuperpixelHierarchy() function to perform X3 segmentation based on the predefined number of superpixel blocks n. b Superpixel segmentation, denoted as S∈R H×W ;

[0025] Step 5: Let the total number of network iterations be T, let epoch = 1, and define the network's optimal validation set loss as...

[0026] Step 5.1 Image Coding

[0027] Step 5.1.1. Define the feature map obtained by passing X3 through a weight-shared head mapping layer with 1×1 two-dimensional convolution as F3∈R. H×W×B′ B′ represents the number of channels in the feature map;

[0028] Step 5.1.2. Construct a pixel-level structure map based on F3. It has HW nodes v1, and the node matrix The i-th row in F3 represents the features of the i-th pixel.

[0029] Step 5.1.3. Based on the result of superpixel segmentation X3, S∈R H×W Define superpixel-level structure graph It has n nodes v2, and a node matrix. and adjacency matrix

[0030] Step 5.1.4. Let the correlation matrix between the pixel-level structure map and the superpixel-level structure map be Q∈R. HW×n Then the expression for Q is shown in equation (3):

[0031]

[0032] Step 5.1.5. Obtain the results according to equations (4) and (5) respectively. and Where σ is a constant, set to 0.1:

[0033]

[0034]

[0035] Step 5.2. Graph Convolution

[0036] Step 5.2.1 Define the graph updated by graph convolution as follows: So adjacency matrix The update rule is shown in equation (6), where S(·) is the Softmax(·) normalization function. BN(·) is the batch normalization function. Here are the parameters to be learned, ⊙ represents the Hadamard product operation, λ is a learnable constant, and I is the identity matrix.

[0037]

[0038] Step 5.2.2 Figure Node feature matrix According to the updated It can be calculated according to equation (7), where σ is a nonlinear activation function. for The degree matrix, The parameters to be learned are obtained from a simple linear layer:

[0039]

[0040] Step 5.3. Obtain the graph according to formula (8). The embedding features of nodes and edges are denoted as follows: and

[0041]

[0042] Step 5.4. Further strengthen salient relationships using the GTrans module;

[0043] Step 5.4.1. Let for The embedding features of the i-th node in the dataset. for The embedding features of the j-th node in the dataset. For nodes and The strength of the inter-adjacency relationships is used as input to the GTrans module and calculated according to equations (9) and (10) to obtain the composition. of

[0044]

[0045]

[0046] Where * denotes element-wise dot product, × denotes matrix multiplication, and W q W k W v W e W g All are learnable mappings of size R. B′×B′ It is learned from the linear layer Linear(·), where ψ(·) is a nonlinear activation function;

[0047] Step 5.4.2. Further update the node feature matrix according to equation (11). The updated record is FFN(·) includes Linear(·), ReLU(·), and Dropout:

[0048]

[0049] Step 5.5. For According to formula (12), the graph is decoded to obtain F4′∈R. HW×B′ and convert to R H×W×B′ , denoted as F4:

[0050]

[0051] Step 5.6. Utilize the gating change information fusion unit to organically connect and integrate multi-temporal characteristics;

[0052] Step 5.6.1. Define the feature map F1∈R obtained after passing X1 and X2 through a weight-shared head mapping layer with 1×1 two-dimensional convolutions. H×W×B′ F2∈R H×W×B′ ;

[0053] Step 5.6.2. Combine F1 and F2 to obtain F i ∈R H×W×2B′ ;

[0054] Step 5.6.3. Combine F3 and F4 to obtain F c ∈R H×W×2B′ ;

[0055] Step 5.6.4. Calculate the variation characteristic F according to formulas (13) and (14). g ∈R H×W×2B′ Where RG is the reset gate, UG is the update gate, S is the Sigmoid(·) activation function, Split(·) is the split operation, and Conv 1×1 (·) represents a 2D convolution operation with a 1×1 kernel, Concat(·) represents a concatenation operation, and Tanh(·) represents the hyperbolic tangent function.

[0056] RG, UG = S(Split(Conv) 1×1 (Concat(F i ,Fc)))) (13);

[0057]

[0058] Step 5.7. Transfer F g The input is fed into the tail mapping layer defined as in equation (15) to obtain pred′∈R. H×W×2 Conv 3×3 (·) represents a 2D convolution operation with a 3×3 kernel, and its dimensions are converted to R. HW×2 The training set detection result pred∈R is obtained. HW×2 ;

[0059] pred′=Softmax(Linear(ReLU(Conv 3×3 (F g )))) (15);

[0060] Step 6. Calculate the network loss according to formula (16), where, m1 is the training set label encoding map obtained in step 3.4, and m2 is the training set label mask map obtained in step 3.5;

[0061]

[0062] Step 7. Train the network parameters using the Adam optimizer based on the loss;

[0063] Step 8. When the epoch is a multiple of 5, i.e., epoch%5==0, validate the network based on the network's encoding of the validation set label obtained in step 3.4. The validation set label mask image m2 obtained in step 3.6 is used to obtain the validation set detection result pred. val And calculate the network validation set loss according to formula (17);

[0064]

[0065] Step 9. Determine the current loss. val Is it less than If it is less than, save the current loss. val As Save the network parameter Θ at the same time; otherwise, no action is required.

[0066] Step 10. Let epoch = epoch + 1. If epoch ≤ T, return to step 5.1 and continue training the network until epoch > T, at which point the network training is complete.

[0067] Step 11. Input the test set into Θ to obtain the test set detection result pred test ∈R HW×2 Use the torch.argmax() function to get the pred test Find the maximum index of each row and transform the result to R. H×W Dimensions are determined to obtain the final test results.

[0068] This invention addresses the problems of insufficient sample size and inadequate perception of distant ground features in existing hyperspectral change detection frameworks. It constructs a topological graph with a semi-supervised learning mode, propagating labels to unlabeled samples and reducing the network's dependence on a large number of training samples. Simultaneously, a graph transformer module with enhanced saliency relationships applies the transformer to a graph structure with superpixels as nodes, enabling the capture of change information across the entire image while breaking through the traditional definition of adjacency relationships under low computational cost. Furthermore, a gated change information fusion unit effectively injects the change features guided by the graph transformer module with enhanced saliency relationships into the original bi-temporal stitched features, achieving the joint integration of multi-temporal information. Experimental results show that, at a sampling rate of 1%, the overall detection accuracy of this invention on the River, Farmland, and USA public datasets reaches 97.49%, 97.91%, and 95.97%, respectively, with Kappa coefficients of 83.66%, 94.92%, and 88.10%, respectively. Attached Figure Description

[0069] Figure 1 This is a network structure diagram of an embodiment of the present invention.

[0070] Figure 2 This is a diagram illustrating the graph encoding structure of an embodiment of the present invention.

[0071] Figure 3 This is a structural diagram of the graph transformer module for enhancing significant relationships according to an embodiment of the present invention.

[0072] Figure 4 This diagram illustrates a comparison of the change detection results of the present invention with those of the SpectralFormer, CSDBF, ML-EDAN, CSANet, SSFTT, and MSDFFN methods on the River dataset.

[0073] Figure 5This diagram illustrates a comparison of the change detection results of the Farmland dataset with the SpectralFormer method, CSDBF method, ML-EDAN method, CSANet method, SSFTT method, and MSDFFN method, based on an embodiment of the present invention.

[0074] Figure 6 This diagram illustrates a comparison of the change detection results of the present invention with those of the SpectralFormer, CSDBF, ML-EDAN, CSANet, SSFTT, and MSDFFN methods on the USA dataset. Detailed Implementation

[0075] The technical solution of this invention is: a multi-temporal information joint hyperspectral change detection method based on graph transformer guidance, such as... Figure 1 The steps shown are as follows:

[0076] Step 1. Denote the hyperspectral image X1∈R at time T1. H×W×C T2 time-phase hyperspectral image X2∈R H×W×C The differential hyperspectral image X3∈R is obtained according to formula (1). H×W×C Where H is the image height, W is the image width, and C is the number of image channels:

[0077] X3 = |X2 - X1| (1);

[0078] Step 2. Preprocess X1, X2, and X3: Convert the dimensions of X1, X2, and X3 to R. HW×C Where HW = H × W, the data with transformed dimensions is standardized using the encapsulated data standardization method sklearn.preprocessing.StandardScaler() to conform to a normal distribution. The standardized data are denoted as follows: and

[0079] Step 3. Based on the label graph g∈R H×W Construct the label encoding map and label mask map in the training set, validation set and test set;

[0080] Step 3.1. Randomly select 1% of the points in g as training data, and generate a dataset of size R. H×W The training set label image z1 is obtained by using a two-dimensional array z1 consisting entirely of zeros. When the point at position (i, j) is a changing pixel, z1(i, j) is 1; when the point at position (i, j) is a non-changing pixel, z1(i, j) is 2.

[0081] Step 3.2. Then, randomly select the same number of points as the training data from the remaining points as validation data, and generate a dataset of size R. H×W The two-dimensional array z2, consisting entirely of zeros, is set to 1 when the point at position (i, j) is a changing pixel and to 2 when the point at position (i, j) is a constant pixel, thus obtaining the validation set label image z2.

[0082] Step 3.3. The remaining points in g are the test data, generating a dataset of size R. H×W The test set label image z3 is obtained by using a two-dimensional array z3 consisting entirely of zeros. When the point at position (i, j) is a changing pixel, z3(i, j) is 1; when the point at position (i, j) is a non-changing pixel, z3(i, j) is 2.

[0083] Step 3.4. Perform dummy variable encoding on z1, z2, and z3 sequentially, that is, encode the 0 point at position (i, j) in the graph as 00, the 1 point at position (i, j) in the graph as 10, and the 2 point at position (i, j) in the graph as 01, and further convert their dimensions to R. HW×2 The training set label encoding maps will be obtained respectively. Validation set label encoding diagram and test set label encoding graph

[0084] Step 3.5. Generate a space with size R HW×2 Given a two-dimensional array m1 containing only 1s, determine... If all pixel values ​​in the i-th row are 0, then all pixel values ​​in the i-th row of m1 are changed to 0 to obtain the training set label mask image m1.

[0085] Step 3.6. Generate a space with size R HW×2 Given a two-dimensional array m2 consisting entirely of 1s, determine... If all pixel values ​​in the i-th row are 0, then modify all pixel values ​​in the i-th row of m2 to 0 to obtain the verification set label mask image m2.

[0086] Step 3.7. Generate a space with size R HW×2 Given a two-dimensional array m3 containing only 1s, determine... If all pixel values ​​in the i-th row are 0, then modify all pixel values ​​in the i-th row of m3 to 0 to obtain the test set label mask image m3.

[0087] Step 4. Superpixel segmentation X3

[0088] Step 4.1. Normalize the value of X3 to the interval [-1, 1], apply a Gaussian filter with a window size of 5×5, and then convert its size to R.HW×C The result is denoted as X3′;

[0089] Step 4.2. Call the encapsulated PCA() function to perform principal component analysis on X3′, and take the first three principal components to reduce the dimensionality of X3′. The result is denoted as X3′. a ;

[0090] Step 4.3. Process X3 according to formula (2). a And convert its value to uint8 type, and the size to R. H×W×3 The result is denoted as P;

[0091]

[0092] Step 4.4. Perform histogram equalization on each channel of P and enhance its contrast using a filtering function. The result is denoted as X3. b ;

[0093] Step 4.5. Call the encapsulated SuperpixelHierarchy() function to perform X3 segmentation based on the predefined number of superpixel blocks n. b Superpixel segmentation, denoted as S∈R H×W ;

[0094] Step 5: Let the total number of network iterations be T, let epoch = 1, and define the network's optimal validation set loss as...

[0095] Step 5.1 Image coding, specifically as follows: Figure 2 As shown:

[0096] Step 5.1.1. Define the feature map obtained by passing X3 through a weight-shared head mapping layer with 1×1 two-dimensional convolution as F3∈R. H×W×B′ B′ represents the number of channels in the feature map;

[0097] Step 5.1.2. Construct a pixel-level structure map based on F3. It has HW nodes v1, and the node matrix The i-th row in F3 represents the features of the i-th pixel.

[0098] Step 5.1.3. Based on the result of superpixel segmentation X3, S∈R H×W Define superpixel-level structure graph It has n nodes v2, and a node matrix. and adjacency matrix

[0099] Step 5.1.4. Let the correlation matrix between the pixel-level structure map and the superpixel-level structure map be Q∈R.HW×n Then the expression for Q is shown in equation (3);

[0100]

[0101] Step 5.1.5. Obtain the results according to equations (4) and (5) respectively. and Where σ is a constant, set to 0.1;

[0102]

[0103] Step 5.2. Graph Convolution

[0104] Step 5.2.1 Define the graph updated by graph convolution as follows: So adjacency matrix The update rule is shown in equation (6), where S(·) is the Softmax(·) normalization function. BN(·) is the batch normalization function. The parameters to be learned are obtained using a linear layer, where ⊙ represents the Hadamard product operation, λ is a learnable constant, and I is the identity matrix.

[0105]

[0106] Step 5.2.2 Figure Node feature matrix According to the updated It can be calculated according to equation (7), where σ is a nonlinear activation function, and the LeakyReLU(·) function is used here. for The degree matrix, The parameters to be learned are obtained from a simple linear layer:

[0107]

[0108] Step 5.3. Obtain the graph according to formula (8). The embedding features of nodes and edges are denoted as follows: and

[0109]

[0110] Step 5.4. Further strengthen salient relationships using the GTrans module, specifically as follows: Figure 3 As shown;

[0111] Step 5.4.1. Let for The embedding features of the i-th node in the dataset. for The embedding features of the j-th node in the dataset. For nodes and The strength of the inter-adjacency relationships is used as input to the GTrans module and calculated according to equations (9) and (10) to obtain the composition. of

[0112]

[0113]

[0114] Where * denotes element-wise dot product, × denotes matrix multiplication, and W q W k W v W e W g All are learnable mappings of size R. B′×B′ It is learned from the linear layer Linear(·), where ψ(·) is a non-linear activation function, and Sigmoid(·) is used here.

[0115] Step 5.4.2. Further update the node feature matrix according to equation (11). The updated record is FFN(·) includes Linear(·), ReLU(·), and Dropout;

[0116]

[0117] Step 5.5. For According to formula (12), the graph is decoded to obtain F4′∈R. HW×B′ and convert to R H×W×B′ , denoted as F4;

[0118]

[0119] Step 5.6. Utilize the gating change information fusion unit to organically connect and integrate multi-temporal characteristics;

[0120] Step 5.6.1. Define the feature map F1∈R obtained after passing X1 and X2 through a weight-shared head mapping layer with 1×1 two-dimensional convolutions. H×W×B′ F2∈R H×W×B′ ;

[0121] Step 5.6.2. Combine F1 and F2 to obtain F i ∈R H×W×2B′ ;

[0122] Step 5.6.3. Combine F3 and F4 to obtain FC ∈R H×W×2B′ ;

[0123] Step 5.6.4. Calculate the variation characteristic F according to formulas (13) and (14). g ∈R H×W×2B′ Where RG is the reset gate, UG is the update gate, S is the Sigmoid(·) activation function, Sokut(·) is the split operation, and Conv 1×1 (·) is a two-dimensional convolution operation with a kernel size of 1×1, Concat(·) is a concatenation operation, and Tanh(·) is a hyperbolic tangent function;

[0124] RG, UG = S(Split(Conv) 1×1 (Concat(F i ,Fc)))) (13);

[0125]

[0126] Step 5.7. Transfer F g The input is fed into the tail mapping layer defined as in equation (15) to obtain pred′∈R. H×W×2 Conv 3×3 (·) represents a 2D convolution operation with a 3×3 kernel, and its dimensions are converted to R. HW×2 The training set detection result pred∈R is obtained. HW×2 ;

[0127]

[0128] Step 6. Calculate the network loss according to formula (16), where, m1 is the training set label encoding map obtained in step 3.4, and m2 is the training set label mask map obtained in step 3.5;

[0129]

[0130] Step 7. Train the network parameters using the Adam optimizer based on the loss;

[0131] Step 8. When the epoch is a multiple of 5, i.e., epoch%5==0, validate the network based on the network's encoding of the validation set label obtained in step 3.4. The validation set label mask image m2 obtained in step 3.6 is used to obtain the validation set detection result pred. val And calculate the network validation set loss according to formula (17);

[0132]

[0133] Step 9. Determine the current loss. val Is it less than If it is less than, save the current loss. val As Save the network parameter Θ at the same time; otherwise, no action is required.

[0134] Step 10. Let epoch = epoch + 1. If epoch ≤ T, return to step 5.1 and continue training the network until epoch > T, at which point the network training is complete.

[0135] Step 11. Input the test set into Θ to obtain the test set detection result pred test ∈R HW×2 Use the torch.argmax() function to get the Pred test Find the maximum index of each row and transform the result to R. H×W Dimensions are determined to obtain the final test results.

[0136] To verify the effectiveness of this invention, experiments were conducted using the publicly available River dataset, Farmland dataset, and USA dataset as examples. Overall accuracy (OA) and Kappa coefficient were used as objective indicators to evaluate the change detection results. The evaluation results of this invention were compared with those of the SpectralFormer, CSDBF, ML-EDAN, CSANet, SSFTT, and MSDFFN methods, where the number of superpixel segments used in this invention was 9000, 10000, and 11000, respectively. The results are as follows: Figures 4-6 As shown in Table 1.

[0137] Table 1 Comparison of change detection accuracy for River, Farmland, and USA datasets

[0138]

[0139] As shown in Table 1, for the River dataset, the OA results obtained by this invention are improved by 0.71%, 1.52%, 2.13%, 2.29%, 1.63%, and 1.62% respectively compared to the SpectralFormer, CSDBF, ML-EDAN, CSANet, SSFTT, and MSDFFN methods; the Kappa coefficient results are improved by 4.25%, 8.68%, 15.28%, 15.55%, 9.91%, and 13.92% respectively compared to the SpectralFormer, CSDBF, ML-EDAN, CSANet, SSFTT, and MSDFFN methods; for the Farmland dataset, the OA results obtained by this invention are improved by 1.10%, 1.28%, 1.12%, and 1.41% respectively compared to the SpectralFormer, CSDBF, ML-EDAN, CSANet, SSFTT, and MSDFFN methods. The Kappa coefficients were improved by 2.62%, 3.07%, 2.69%, 3.42%, 5.89%, and 3.62% respectively compared to the SpectralFormer, CSDBF, ML-EDAN, CSANet, SSFTT, and MSDFFN methods. For the USA dataset, the OA results obtained by this invention were improved by 2.45%, 2.45%, and 1.54% respectively compared to the SpectralFormer, CSDBF, ML-EDAN, CSANet, SSFTT, and MSDFFN methods. The N method, CSANet method, SSFTT method, and MSDFFN method improved performance by 0.99%, 1.63%, 2.08%, 2.11%, 2.57%, and 1.10%, respectively; the Kappa coefficient results were improved by 2.51%, 3.74%, 6.27%, 5.88%, 5.31%, and 3.40%, respectively, compared to the SpectralFormer method, CSDBF method, ML-EDAN method, CSANet method, SSFTT method, and MSDFFN method.

[0140] Figure 4The images show the change detection results of different methods on the River dataset. (a) shows the change detection results of the SpectralFormer method, with an overall accuracy of 96.78%; (b) shows the change detection results of the CSDBF method, with an overall accuracy of 95.97%; (c) shows the change detection results of the ML-EDAN method, with an overall accuracy of 95.36%; (d) shows the change detection results of the CSANet method, with an overall accuracy of 95.20%; (e) shows the change detection results of the SSFTT method, with an overall accuracy of 95.86%; (f) shows the change detection results of the MSDFFN method, with an overall accuracy of 95.87%; (g) shows the change detection results of the present invention, with an overall accuracy of 97.49%; and (h) shows the ground-truth change detection map.

[0141] Figure 5 The images show the change detection results of different methods on the Farmland dataset. (a) shows the change detection results of the SpectralFormer method, with an overall accuracy of 96.81%; (b) shows the change detection results of the CSDBF method, with an overall accuracy of 96.63%; (c) shows the change detection results of the ML-EDAN method, with an overall accuracy of 96.79%; (d) shows the change detection results of the CSANet method, with an overall accuracy of 96.50%; (e) shows the change detection results of the SSFTT method, with an overall accuracy of 95.46%; (f) shows the change detection results of the MSDFFN method, with an overall accuracy of 96.37%; (g) shows the change detection results of the present invention, with an overall accuracy of 97.91%; and (h) shows the ground-truth change detection map.

[0142] Figure 6 The images show the change detection results of different methods on the USA dataset. (a) shows the change detection results of the SpectralFormer method, with an overall accuracy of 94.98%; (b) shows the change detection results of the CSDBF method, with an overall accuracy of 94.34%; (c) shows the change detection results of the ML-EDAN method, with an overall accuracy of 93.89%; (d) shows the change detection results of the CSANet method, with an overall accuracy of 93.86%; (e) shows the change detection results of the SSFTT method, with an overall accuracy of 93.40%; (f) shows the change detection results of the MSDFFN method, with an overall accuracy of 94.87%; (g) shows the change detection results of the present invention, with an overall accuracy of 95.97%; and (h) shows the ground-truth change detection map.

[0143] from Figure 4 , Figure 5 and Figure 6 As can be seen, the present invention has the highest consistency with the ground-truth change detection map. It can detect large-area change areas while also capturing small-scale changes in ground features. The present invention can obtain more accurate change detection results than the SpectralFormer method, CSDBF method, ML-EDAN method, CSANet method, SSFTT method, and MSDFFN method.

[0144] Summary Table 1 Figure 4 , Figure 5 , Figure 6 The comparison results show that by introducing a graph transformer module that enhances significant relationships and a gated change information fusion unit, the present invention effectively improves the accuracy of hyperspectral change detection.

Claims

1. A multi-temporal information joint hyperspectral change detection method based on graph transformer guidance, characterized by The following steps are performed in sequence: Step 1. Hyperspectral image at T1 time phase Step 2. Hyperspectral image at T2 time phase Step 3. Difference hyperspectral image is obtained according to formula (1) wherein H is image height, W is image width, C is image channel number: (1)； Step 2. Preprocessing of , and : converting the dimensions of , , to , where , and normalizing the data of the converted dimensions using the packaged data normalization method to make them conform to the normal distribution. The normalized data are sequentially denoted as , and ; Step 3. Construct label encoding map and label mask map in training set, validation set and test set Construct label encoding map and label mask map in training set, validation set and test set Step 3.

1. Random sampling Using 1% of the points as training data, a dataset of size [size missing] is generated. A two-dimensional array of all zeros When position When the point on the screen is a changing pixel, =1; when position When the points on the graph are constant pixels, The value is 2, thus obtaining the training set label map. ; Step 3.

2. Randomly select the same number of points as the training data from the remaining points as the validation data, generate a two-dimensional array of size of all 0s , when the point at position is a changing pixel, is 1; when the point at position is an unchanged pixel, is 2, to obtain the validation set label map ; Step 3.

3. The rest of the points in the test data are set to 0. A 2D array of size of all 0s is created. When the point at position is a changed pixel, is 1; when the point at position is an unchanged pixel, is 2, resulting in the test set label map ; Step 3.

4. To , and , dummy variable encoding is performed in sequence, i.e. 0 points in the position of the figure are encoded as 00, 1 points in the position of the figure are encoded as 10, and 2 points in the position of the figure are encoded as 01, and further their sizes are converted to , the training set label encoding figure , the validation set label encoding figure and the test set label encoding figure will be obtained respectively; Step 3.

5. Generate a two-dimensional array of size with all 1s , judge the value of each row of pixels in , if all the pixel values in the th row are 0, modify all the pixel values in the th row of to 0 to obtain the training set label mask image ; Step 3.

6. Generate a two-dimensional array of size of all 1s , judge the value of each row of pixels in , if all the pixel values in the first row are 0, modify all the pixel values in the first row of to 0 to obtain the verification set label mask image ; Step 3.

7. Generate a size of A two-dimensional array of all 1s ,judge The value of each pixel in the middle, if the first row... If all pixel values ​​in a row are 0, then... The Middle Change all pixel values ​​in the row to 0 to obtain the test set label mask image. ; Step 4. Superpixel Segmentation X 3 Step 4.

1. Normalize the values of to the interval [0, 1] and apply a Gaussian filter with window size 5x5 and convert the size to , resulting in ; and ; Step 4.

2. Call the encapsulated PCA() function to implement principal component analysis on and take the first three principal components to implement dimensionality reduction on , and the result is recorded as ; Step 4.

3. Treat as formula (2) and convert its value to uint8 type, size to , result is recorded as : (2) ； Step 4.

4. Perform a histogram equalization operation on each channel of and enhance its contrast with a filter function, resulting in ; Step 4.

5. Call the encapsulated SuperpixelHierarchy() function to divide the superpixel into blocks according to the predefined number of superpixel segments. To achieve Superpixel segmentation, denoted as ; Step 5 Set the total number of iterations of the network , let , define the optimal validation set loss of the network as ; Step 5.

1. Map coding Step 5.1.

1. Define pair The feature map obtained after the weight-shared head-mapping layer with 1x1 two-dimensional convolution is , represents the number of channels of the feature map; Step 5.1.

2. Based on Constructing the pixel-level structure map It has nodes , and the node matrix , The first row in represents the characteristics of the first pixel in Step 5.1.

3. Defining the superpixel level graph based on the results of the superpixel segmentation ;​​​​​​ Step 5.1.

4. Set the association matrix between the pixel-level structure map and the superpixel-level structure map as Then The expression of is shown as equation (3): (3)； Step 5.1.

5. Obtained according to formula (4) and formula (5) respectively and wherein is a constant set to 0.1 : (4)； (5)； Step 5.

2. Map convolution Step 5.2.1 defines the graph updated by graph convolution as Then the adjacency matrix of The update rule is shown in equation (6), where, is a normalization function, , is a batch normalization function, is a parameter to be learned, represents the Hadamard product operation, is a learnable constant, is an identity matrix: (6) Step 5.2.2 Figure node feature matrix of According to the updated The result can be calculated as in equation (7), where, is a non-linear activation function, is the degree matrix of , , is the parameter to be learned, obtained by a simple linear layer: (7) ； Step 5.

3. Obtain the graph from formula (8) The embedding features of the nodes and edges are denoted as and respectively. (8)； Step 5.

4. Further strengthen the salient relationship by using the GTrans module; Step 5.4.

1. Let for The Middle Embedding features of each node for The Middle Embedding features of each node For nodes and The strength of the inter-adjacency relationships is used as input to the GTrans module and calculated according to equations (9) and (10) to obtain the composition. of ; (9) ； (10)； wherein, denotes element-wise multiplication, denotes matrix multiplication, , , , , are learnable mappings of size are learned by linear layers , is a non-linear activation function; Step 5.4.

2. Further update the node feature matrix as formula (11) , the updated one is denoted as where contains , , Dropout: (11) ； Step 5.

5. To decode the graph according to equation (12) to obtain and convert to , denoted as : (12)； Step 5.

6. Organic combination of multi-temporal features by using the gating change information fusion unit; Step 5.6.

1. Define pair and The feature map after the weight-shared head mapping layer with 1x1 two-dimensional convolution is , ; Step 5.6.

2. Splicing and to give ; Step 5.6.

3. Splicing and to give ; Step 5.6.

4. Calculate the change characteristics according to formulas (13) and (14). ,in, To reset the door, To update the door, for Activation function For separation operation, The convolution kernel is Two-dimensional convolution operations of size, For splicing operations, For hyperbolic tangent function: (13)； (14)； Step 5.

7. Input to define the tail mapping layer as formula (15) to obtain , wherein , wherein is a two-dimensional convolution operation with a convolution kernel of size , and convert its size to , to obtain the training set detection result (15)； Step 6. Calculate the network loss according to formula (16), wherein, is the training set label encoding map obtained at step 3.4, is the training set label mask map obtained at step 3.5; (16)； Step 7. Train the network parameters using Adam optimizer according to loss = -log(p) Step 8. When When it is a multiple of 5, that is At that time, the network is verified based on the verification set label encoding diagram obtained in step 3.

4. and the verification set label mask image obtained in step 3.6 The validation set detection results were obtained. And calculate the network validation set loss according to formula (17); (17)； Step 9. Determine if the current is less than , if so, save the current as , while saving the network parameters , otherwise no action is required; Step 10. Let , if , return to step 5.1 to continue training the network until , the network training is completed; Step 11. Input the test set into... The test set detection results were obtained. Use the torch.argmax() function to obtain Find the maximum index of each row and transform the result to... Dimensions are determined to obtain the final test results.

Images