Multi-scale GCN and DCN hyperspectral image classification method based on image reconstruction

Abstract

The invention relates to the technical field of remote sensing image processing and computer vision, and discloses a multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction, which comprises the following steps of: adaptively capturing irregular spatial features by combining a dynamic spectral feature extraction module (DSFEM) with deformable convolution (DCN); a multi-scale space-spectrogram convolution module (MSSGCM) is used for extracting global features under different segmentation scales, and a graph structure is dynamically adjusted through a graph reconstruction module (GSRM-GL) so as to accurately model the relation between super-pixel nodes. Meanwhile, a spatial spectrum joint loss function (SSJL) fuses cross entropy loss, spatial total variation loss and spectrum total variation loss, and optimizes classification precision and feature coherence. According to the multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction, the classification precision on data sets such as Indian Pines and University of Pavia is remarkably improved, high robustness is still kept under the condition of few samples, and the problems that feature extraction is limited and the graph structure is solidified in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the fields of remote sensing image processing and computer vision technology, and in particular to a multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction. Background Art

[0002] Hyperspectral remote sensing technology enables the integrated observation of spatial and spectral information. Therefore, hyperspectral images (HSI) obtained through hyperspectral remote sensing technology are three-dimensional image data that contain both spectral and spatial information. By comprehensively analyzing spatial and spectral information, land cover classification can be accurately identified, which is the primary task of hyperspectral image classification.

[0003] In the field of hyperspectral image (HSI) classification, convolutional neural networks (CNNs) and graph convolutional networks (GCNs) are two widely used mainstream technologies. CNNs, with their fixed convolution kernel structure, can effectively extract pixel-level features and local spatial dependencies. Graph convolutions, on the other hand, typically operate on superpixel structures, extracting irregular superpixel-level features and modeling long-range relationships.

[0004] Although combining CNN and GCN can significantly improve classification results, convolutional neural networks (CNNs) have difficulty capturing the irregular spatial coverage relationships in hyperspectral images, resulting in limitations in feature extraction. Secondly, while graph convolutional networks (GCNs) can handle irregular graph structures, they typically perform convolution operations based on a fixed graph structure. Such graph structures often rely on initial graph construction, which often fails to accurately reflect the geometric and spectral similarities between superpixel nodes, resulting in the central node being unable to perceive important features in the graph structure. Therefore, traditional methods are limited in classification accuracy and stability, and are particularly prone to misclassification when dealing with scenes with complex ground features and irregular spatial information.

[0005] Therefore, existing hyperspectral image classification technology has obvious deficiencies in extracting irregular ground features and accurately modeling graph structures:

[0006] 1. Although convolutional neural networks (CNNs) can effectively extract local features, their fixed convolution kernel structure makes it difficult to capture the irregular spatial coverage relationships in hyperspectral images and is prone to overfitting when the sample size is limited.

[0007] 2. Although graph convolutional networks (GCNs) have achieved certain advantages in superpixel structures, traditional methods mostly rely on preset fixed graph structures, which make it difficult to fully express the complex relationships between nodes. As a result, it is difficult for central nodes to perceive key features in complex neighborhoods, thus affecting the accuracy and robustness of classification.

[0008] Therefore, it is necessary to provide a multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction, which can fully utilize the multi-scale feature information in irregular spatial distribution and dynamically adjust the graph structure to more accurately reflect the relationship between nodes, thereby improving the accuracy and robustness of further hyperspectral image classification. Summary of the Invention

[0009] The present invention provides a multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction, which aims to effectively solve the problem that the existing technology is difficult to capture irregular spatial features and the initial graph cannot accurately model the superpixel node relationship. Figure 1 As shown in the figure, this method extracts local features by adjusting the shape of the convolution kernel and combines multi-scale graph convolution to capture global features at different segmentation scales. By constructing a graph reconstruction module, the central node can effectively perceive the key features in the graph structure and more accurately reflect the geometric and spectral similarities between nodes, thereby significantly improving classification accuracy, continuity of spatial-spectral information, and robustness under small sample conditions.

[0010] The technical solutions of the present invention are as follows:

[0011] The multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction provided by the present invention includes the following steps:

[0012] 1. Preprocessing Hyperspectral Images

[0013] The input hyperspectral image is represented as X∈R H×W×B , where H represents the height of the image, W represents the width, and B represents the spectral dimension. Use principal component analysis (PCA) to reduce the dimensionality of the spectral data to reduce information redundancy and obtain the band data after dimensionality reduction

[0014] 2. Dynamic Spectral Feature Extraction Module (DSFEM)

[0015] 1. First, input hyperspectral image data After a 1x1 convolutional layer, it is used to keep the spatial dimension unchanged while reducing the computational cost and extracting the deep spectral features of the image.

[0016]

[0017] Where: X(i,j,n) represents the nth spectral channel value at position (i,j) in the input feature map X, is the weight of the 1x1 convolution kernel, F 1×1 ∈R H×W×CIt is the output feature map after the 1x1 convolution layer, where C represents the number of output channels and * represents the convolution operation.

[0018] 2. Perform deformable DCN convolution operation

[0019] ①Generate the initial sampling point coordinate set

[0020] like Figure 2 As shown in the feature map F 1×1 Generate a specified number N of initial sampling point coordinate sets P n ={(p n,x ,p n,y )|n=1,2,...,N}, these sampling point coordinates represent the position of each sampling point in the convolution kernel, where p n,x and p n,y Represents the coordinates of the nth sampling point in the x and y directions respectively.

[0021] ② Learning the offset and calculating the coordinates of the new sampling points

[0022] In order to adapt to the different shapes and regions of hyperspectral image objects, the network generates an offset ΔP for each sampling point through a trainable two-dimensional convolutional layer to dynamically adjust the position of the sampling point. The offset is generated as follows: ΔP = W 3×3 *F 1×1 , where W 3×3 is a 3x3 convolution kernel used to generate the offset, and the number of output channels is 2N, where N represents the number of sampling points, so the output contains the displacement of each sampling point in the x and y directions. The offset set ΔP contains the displacement pair of each sampling point: ΔP∈{(Δp n,x ,Δp n,y )|n=1,2,...,N}, where Δp n,x and Δp n,y Represents the displacement of the nth sampling point in the x and y directions respectively.

[0023] ③The generated offset ΔP is added to the initial sampling point coordinate P0 and the convolution kernel relative position P n To calculate the new sampling point coordinates P', the formula is: P' = P0 + P n +ΔP, where: P0 represents the reference coordinate grid, which defines the fixed position of all pixels in the image; P n Represents the offset of each sampling point in the convolution kernel relative to the reference coordinates. ΔP is the offset learned by convolution.

[0024] ④ Bilinear interpolation to calculate eigenvalues

[0025] At the new sampling point coordinate P′, bilinear interpolation is applied to calculate the eigenvalue to maintain the continuity and spatial consistency of the spectral information and reduce the information loss during the resampling process. The calculation formula is:

[0026] x offset =g lt X(q lt )+g rb X(q rb )+g lb X(q lb )+g rt X(q rt )

[0027] g lt =(1+(q lt,x -p x ′))·(1+(q lt,y -p′ y ))

[0028] g rb =(1-(q rb,x -p x ′))·(1-(q rb,y -p′ y ))

[0029] g lb =(1+(q lb,x -p x ′))·(1-(q lb,y -p′ y ))

[0030] g rt =(1-(q rt,x -p x ′))·(1+(q rt,y -p′ y ))

[0031] Where: q lt ,q rb ,q lb ,q rt Represents the coordinates of the four integer sampling points at the upper left, lower right, lower left and upper right corners respectively; X(q lt ),X(q rb ),X(q lb ),X(q rt ) represent the feature values ​​of these four sampling points in the input feature map.

[0032] g lt ,g rb ,g lb ,g rt is the interpolation weight.

[0033] ⑤ Rearrangement and column convolution operations

[0034] Resampled feature map x offset ∈R C×H×W×N Rearrange to x offset ∈R C×(H×W)×N , use a two-dimensional convolution kernel of size Nx1 to perform column convolution to extract the spectral change characteristics in the vertical direction. The column convolution is calculated as follows:

[0035]

[0036] Where: Y i，j is the value of the output feature map at position (i, j);

[0037] W n is the weight of the column convolution kernel, which represents the weight of different positions in the column direction;

[0038] x offset (i,j,n) represents the resampled eigenvalue at position (i,j).

[0039] ⑥Batch normalization and activation function

[0040] The output Y of the column convolution is processed by batch normalization (Norm) and nonlinear activation function (SiLU) to generate the intermediate feature map F int :

[0041] F int (i,j)=SiLU(Norm(Y i,j ))

[0042] 3. The second 1x1 convolutional layer fuses features

[0043] After completing the DCN spatial feature extraction, the second 1x1 convolutional layer fuses the inter-band information and integrates the information of different spectral regions to extract the discriminative spectral features to obtain the final pixel-level output feature H pf :

[0044]

[0045] in is the weight of the second 1x1 convolution kernel, and C represents the number of final output channels.

[0046] 3. Multiscale Spectral-Spatial Graph Convolution Module (MSSGCM)

[0047] 1. Use the Boruvka algorithm to divide the hyperspectral image after dimensionality reduction into five layers of superpixel structures with different segmentation scales through progressive merging: represents the superpixel set of layer l, m l Represents the number of superpixels in the lth layer. The number of superpixels decreases as the scale increases (m l >m l+1 , m 0 =H*W), where H and W represent the height and width of the input image respectively.

[0048] 2. Construct superpixel relationship matrix, matrix The relationship between superpixels in layer l and layer l+1 can be modeled as follows:

[0049]

[0050] Q l+1 The i-th row and j-th column represent the superpixel area of ​​layer l Whether it has participated in merging into the superpixel area of ​​the l+1 layer

[0051] 3. Graph convolution input feature construction, for the multi-scale GCN module, first use the matrix Constructing the input features of the first layer of graph convolution Among them, B * is the spectral dimension of the input image, Each line and S 1 Each of Corresponding:

[0052]

[0053] in Represents the column-wise matrix Q 1 Normalize (e.g. ), Reshape() means mapping the input from a three-dimensional tensor to a two-dimensional matrix (H×W×B * →HW×B * ).

[0054] 4. Construct an adjacency matrix. Based on the spatial similarity principle and the selection criteria for neighbor nodes, the graph structure is defined as follows:

[0055]

[0056] Among them A 1 The i-th row and j-th column in the y-axis represents the superpixel area of ​​the l-th layer. and Whether they are adjacent in the spatial dimension (i.e., whether there are common edges).

[0057] 5. Multi-scale graph convolution calculation, for A 1 Performing row-wise averaging operation yields (in ),use Perform graph convolution to extract superpixel features at the first segmentation scale

[0058]

[0059] in Represents the convolution parameters. In order to fully explore the influence of the features of different topological structures on the classification results, the output features Further split into three sub-feature matrices by channel Then, use Constructing input features of graph convolutions of different scales

[0060]

[0061] Next, use the adjacency matrix: Correspondingly, parallel graph convolution is performed on graph structures of different scales to extract superpixel-level features:

[0062]

[0063]

[0064] W2, W3, and W4 represent the convolution parameters of different layers.

[0065] 6. Multi-scale feature integration, multi-scale features Integration obtains multi-scale super-pixel output features H ms :

[0066]

[0067] 4. Hierarchical Feature Fusion Module (HFFM)

[0068] 1. Input pixel-level features H pf and multi-scale super-pixel features H ms .

[0069] 2. Pixel-level features H pf and super-pixel features H ms Connect along the channel dimension to generate a unified feature representation H i :

[0070] Hi =concat(H pf ,H ms )

[0071] Among them, the collection

[0072] 3. Use depth-wise separable convolution (SSConv) to fused features H i Perform convolution operation to generate further fused features H fuse =SSConv(H i ), the SSConv operation can effectively reduce the computational complexity and number of parameters of the model while fusing spatial and spectral information, thereby improving the efficiency of the model.

[0073] 5. Graph Structure Reconstruction Based on Geometric and Label Information (GSRM-GL)

[0074] 1. Initial construction of super-pixel regions. The Boruvka minimum spanning tree algorithm is used to divide the hyperspectral image into super-pixel regions of multiple scales. Each super-pixel region can be regarded as the result of an unsupervised clustering.

[0075] Each superpixel region is regarded as a collection of similar pixels for subsequent graph structure reconstruction.

[0076] 2. Calculate label information:

[0077] 2.1 As Figure 3 As shown, the feature representation H output by the model is used fuse and feature matrices at different levels Generate multiple cluster centers

[0078] 2.2 For each superpixel region, calculate its cluster center in spatial and spectral dimensions to form feature representations at different levels.

[0079] 2.3 Feature H fuse Divide into labeled sets and unlabeled sets Two parts: Calculate the center of the labeled set for subsequent geometric similarity calculation:

[0080]

[0081] Among them C n Indicates the label node center of the nth category, y vrepresents the true label of the labeled node, J represents the total number of categories, and M(·) represents the mean function.

[0082] 3. Calculate geometric distance

[0083] 3.1 Use cosine distance to measure the directional difference between superpixel centers. The formula is as follows:

[0084]

[0085] in, and are the feature vectors of the superpixel region and label center, respectively.

[0086] 3.2 Use the normalized Euclidean distance to measure the spatial difference of superpixel centers. First, normalize the features and then calculate the normalized Euclidean distance:

[0087]

[0088] 3.3 Combine cosine distance and normalized Euclidean distance to calculate fusion similarity:

[0089]

[0090] Wherein, α1 and α2 are weight coefficients, satisfying α1+α2=1.

[0091] 4. Pseudo-label assignment

[0092] 4.1 For each superpixel region, select the label center with the minimum distance based on the fusion similarity and assign a pseudo label:

[0093]

[0094] Represents the i-th superpixel area in the l-th layer and the center of the nth class The geometric similarity metric is to find the best matching feature between the category center and the current superpixel region by selecting the minimum distance, and assign a pseudo label to each superpixel region in the lth layer.

[0095] 5. Redefinition of Graph Structure

[0096] The adjacency relationship of the graph structure is redefined by combining geometric similarity and label information. The new adjacency matrix is ​​defined as follows:

[0097]

[0098] Among them, spatial similarity and label information jointly determine the connectivity between nodes, ensuring that the graph structure only retains nodes with high similarity.

[0099] Feed the newly constructed graph structure into the original model and retrain the entire model:

[0100]

[0101] in

[0102] 6. Spatial-Spectral Joint Loss Function (SSJL)

[0103] 1. Spatial-spectral joint loss function L total The overall expression is:

[0104] L total =L CE +λ spatial ·L TV,spatial +λ spectral ·L TV,spectral

[0105] Where: L CE is the cross entropy loss, which is used to measure the difference between the model prediction category and the true category; L TV,spatial is the total spatial variation loss, which is used to maintain the smoothness of the image space; L TV,spectral is the total spectral variation loss, which ensures the smoothness of the image in the spectral dimension; spatial and λ spectral It is a weight parameter used to adjust the contribution of different loss items to the total loss to balance the classification accuracy and the coherence of spatial spectral features.

[0106] 2 Calculation of cross entropy loss

[0107] Cross entropy loss L CE It is used to calculate the difference between the predicted category and the true category, which is defined as:

[0108]

[0109] Among them, T is the number of training samples, C is the number of categories, and y n,c is the one-hot encoding of the nth sample, is the model's predicted probability of the cth class for the nth sample. Cross entropy loss is a commonly used loss function in classification tasks, used to guide the model to improve classification accuracy.

[0110] 3 Calculation of total spatial variation loss

[0111] L TV,spatial Used to maintain the smoothness of the image in the spatial dimension, it is defined as:

[0112]

[0113] Among them, H and W are the height and width of the image respectively, x n,i,j Represents the pixel value of the nth sample at the (i, j)th position. The function of this term is to maintain the spatial smoothness of the image by minimizing the difference between adjacent pixels and prevent abnormal changes in the image space.

[0114] 4 Calculation of total spectral variation loss

[0115] L TV,spectral Used to maintain the smoothness of the image in the spectral dimension, it is defined as:

[0116]

[0117] Among them, x n,k Represents the pixel value of the nth sample at spectral channel k. This term minimizes the difference between adjacent spectral channels, maintains the smoothness of spectral information, and ensures a more stable feature expression of the spectral image.

[0118] 7. Classification Prediction

[0119] The reconstructed graph features are input into the multi-layer perceptron (MLP) layer for classification, and the confidence of each category is calculated using the softmax function to obtain the classification probability P of each pixel.

[0120] P=soft max(FCL(H fuse ))

[0121] in, J represents the total number of categories.

[0122] Predict the category to which each pixel belongs, and calculate the overall accuracy OA, average accuracy AA, and Kappa coefficient between the true category and the predicted category to evaluate the effectiveness of the classification method.

[0123] Overall accuracy OA: Overall accuracy is the ratio of the number of samples correctly classified by the classifier in all categories to the total number of samples.

[0124]

[0125] Average precision (AA): used to measure the performance of a classification algorithm in multi-category classification tasks.

[0126] This metric calculates the classification accuracy for each class and then averages these accuracies.

[0127]

[0128] in,

[0129] Kappa coefficient: The Kappa coefficient takes into account the randomness in the classification. Its value ranges from -1 to 1, where 1 means that the classifier's prediction is completely consistent with the actual situation, 0 means that the classifier's prediction result is equivalent to the random prediction result, and when it is a negative value, it means that the consistency between the classifier's prediction result and the actual situation is not as good as random prediction.

[0130]

[0131] Assume that the number of real samples of each class is a1, ...ac, and the number of samples of each class predicted is b1, b2 ..., bc, the total number of samples is b1, b2, ...bc, and the total number of samples is ρ, then

[0132] Compared with related technologies, the multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction provided by the present invention significantly improves classification accuracy and model robustness by dynamically updating graph structure and extracting multi-scale features. Specifically, the present invention has the following technical effects:

[0133] 1. Significantly Improved Classification Accuracy: Through the Dynamic Spectral Feature Extraction Module (DSFEM) and the Parallel Multiscale Spatial-Spectral Graph Convolution Module (MSSGCM), the proposed method extracts features from superpixel structures at different segmentation scales, fully capturing both local and global features of ground object information. Experimental results based on four standard datasets (Indian Pines, University of Pavia, Salinas, and Kennedy Space Center) demonstrate that the proposed method has higher accuracy and generalization ability. Under the same sample conditions, among seven neural network comparison algorithms, the proposed algorithm improves OA by 1.86%-33.5% on the 2% training set of Indian Pines, 1.16%-38.14% on the 0.2% training set of University of Pavia, 3.84%-28.11% on the 5-sample training set of Salinas, and 1.42%-42.88% on the 2% training set of KSC.

[0134] 2. Enhance the robustness of the model: The graph structure reconstruction module (GSRM-GL) of the present invention combines geometric information and label information to dynamically adjust the graph structure, retaining only high-similarity nodes, thereby improving the adaptability of the graph structure to complex ground feature information. Figure 4As shown in the figure, under the condition of few samples, the Indian Pines dataset can still maintain excellent classification results with 1%-2% training samples, and shows high accuracy and high stability in the classification of both overall and small class samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0135] Figure 1 Schematic diagram of parallel multi-scale GCN-DCN hyperspectral image classification network based on geometric information and label information reconstruction;

[0136] Figure 2 In the DCN dynamic sampling process, taking 5 sampling points as an example, deformable convolution generates dynamic offsets, obtains integer eigenvalues ​​through bilinear interpolation, and uses a convolution kernel of size (N, 1) to extract spatial features after rearrangement;

[0137] Figure 3 To reconstruct the process diagram, we take 6 superpixel nodes as an example, representing J different categories, and represent the pseudo labels generated by the 6 superpixel nodes. In the superpixel structure diagram, we use different colors to represent different pseudo labels;

[0138] Figure 4 The following are line graphs of the overall classification accuracy (OA) under different sample sizes: (A) OA accuracy of the Indian Pines dataset under (0.5%, 1%, 2%, 3%, 4%) training samples, (B) OA accuracy of the University of Pavia dataset under (0.1%, 0.2%, 0.3%, 0.4%, 0.5%) training samples, (C) OA accuracy of the Salinas dataset under (3, 5, 7, 9, 11) sample sizes, and (D) OA accuracy of the Kennedy Space Center dataset under (0.5%, 1%, 2%, 3%, 4%) training samples.

[0139] Figure 5 Comparison results of different algorithms for Salinas under 5 training samples: (a) ground truth map, (b) false color image, (c) CNN (OA = 69.66%), (d) SSRN (OA = 86.92%), (e) HybridSN (OA = 90.73%), (f) CEGCN (OA = 92.63%), (g) WFCG (OA = 92.58%), (h) MSSGU (OA = 93.93%), (i) AMGCF (OA = 92.88%), (j) Proposed (OA = 97.77%). DETAILED DESCRIPTION

[0140] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0141] The multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction provided by the present invention includes the following steps:

[0142] Step S1, data preprocessing: preprocess the collected hyperspectral image data and public datasets to prepare for subsequent model training.

[0143] The specific process of step S1 is:

[0144] Step S11: data set division;

[0145] The Salinas dataset consists of hyperspectral images acquired by the U.S. Department of Agriculture (USDA) using an aerial imager (AVIRIS). The images are 512×217 pixels in size and have a spatial resolution of 3.7 meters. The Salinas dataset contains 224 continuous spectral bands, covering a wide range from visible light to near-infrared. The images in the dataset cover 16 different land feature categories, totaling 53,987 samples, including farmland, vegetation, and water bodies. Five samples from each category are randomly selected as training samples, 0.1% of the samples are used as validation samples, and the rest are used as test samples. The following table shows the data partitioning settings for the training, validation, and test sets of the Salinas dataset.

[0146] Step S12: Data processing

[0147] For Salinas hyperspectral data X∈R H×W×B Perform PCA dimensionality reduction processing to obtain the band data after dimensionality reduction The data height is H = 512, the width is W = 217, and the number of spectral channels is B. * =64, and the reduced-dimensional data is sent to two branches in parallel.

[0148] Step S2: Model building: Establish a graph reconstruction-based fusion multi-scale GCN and DCN hyperspectral image classification network.

[0149] Specifically, step S2 includes the following steps:

[0150] Step S21: construct a dynamic spectral feature extraction module (DSFEM) to capture local pixels and enhance adaptability to spectral changes.

[0151] 1. Perform 1x1 convolution on the dimensionality-reduced Salinas data to extract the deep spectral features of the image.

[0152]

[0153] Where: X(i,j,n) represents the nth spectral channel value at position (i,j) in the input feature map X;

[0154] is the weight of the 1x1 convolution kernel, F 1×1 ∈R H×W×C It is the output feature map after the 1x1 convolution layer, where C represents the number of output channels and * represents the convolution operation.

[0155] 2. Use DCN to extract pixel-level features of local irregular areas:

[0156]

[0157] Where: Y i，j is the value of the output feature map at position (i, j);

[0158] W n is the weight of the column convolution kernel, which represents the weight of different positions in the column direction;

[0159] x offset (i,j,n) represents the resampled eigenvalue at position (i,j);

[0160] * indicates a convolution operation.

[0161] After the column convolution operation, batch normalization (Norm) and activation function (SiLU) are used to generate the intermediate feature map F int :

[0162] F int (i,j)=SiLU(Norm(Y i,j ))

[0163] 3. After completing DCN spatial feature extraction, in order to further fuse the information between bands, we pass these spatial features to a 1x1 convolution layer. 1x1 convolution can integrate information from different spectral regions and extract more discriminative spectral features. The final output pixel-level feature is H pf , containing diverse spectral and spatial information.

[0164]

[0165] Step S22: Construct a multi-scale spatial-spectral graph convolution module (MSSGCM) to extract global features of different scales in parallel.

[0166] 1. Use the Boruvka algorithm to divide the dimensionality-reduced Salinas image into five layers of superpixel structures with different segmentation scales through progressive merging: represents the superpixel set of layer l, m lRepresents the number of superpixels in the lth layer. The number of superpixels decreases as the scale increases (m l >m l+1 , m 0 =H*W), where H and W represent the height and width of the input image respectively.

[0167] 2. Construct the Salinas superpixel relationship matrix, matrix The relationship between superpixels in layer l and layer l+1 can be modeled as follows:

[0168]

[0169] Q l+1 The i-th row and j-th column represent the superpixel area of ​​layer l Whether it has participated in merging into the superpixel area of ​​the l+1 layer

[0170] 3. Graph convolution input feature construction, for the multi-scale GCN module, first use the matrix Constructing the input features of the first layer of graph convolution Among them, B * is the spectral dimension of the input image, Each line and S 1 Each of Corresponding:

[0171]

[0172] in Represents the column-wise matrix Q 1 Normalize (e.g. ), Reshape() means mapping the input from a three-dimensional tensor to a two-dimensional matrix (H×W×B * →HW×B * ).

[0173] 4. Construct the adjacency matrix. Based on the definition of spatial similarity and the selection of neighbor nodes, we define the graph structure as follows:

[0174]

[0175] Among them A 1 The i-th row and j-th column in the y-axis represents the superpixel area of ​​the l-th layer. and Whether they are adjacent in the spatial dimension (i.e., whether there are common edges).

[0176] 5 Multi-scale graph convolution calculation. 1 Performing row-wise averaging operation yields (in ),use Perform graph convolution to extract superpixel features at the first segmentation scale

[0177]

[0178] in Represents the convolution parameters. In order to fully explore the influence of the features of different topological structures on the classification results, the output features Further split into three sub-feature matrices by channel Then, use Constructing input features of graph convolutions of different scales

[0179]

[0180]

[0181] Next, use the adjacency matrix: Correspondingly, parallel graph convolution is performed on graph structures of different scales to extract superpixel-level features:

[0182]

[0183] W2, W3, and W4 represent the convolution parameters of different layers.

[0184] 6. Multi-scale feature integration, multi-scale features Integration obtains multi-scale super-pixel output features H ms :

[0185]

[0186] 7. Input Salinas pixel-level feature H pf and Salinas multi-scale super-pixel feature H ms .

[0187] 8. Pixel-level features H pf and super-pixel features H ms Connect along the channel dimension to generate a unified feature representation H i :

[0188] H i =concat(H pf ,H ms )

[0189] Among them, the collection

[0190] 9. Use depth-wise separable convolution (SSConv) to fused features Hi Perform convolution operation to generate further fused features H fuse =SSConv(H i ), the SSConv operation can effectively reduce the computational complexity and number of parameters of the model while fusing spatial and spectral information, thereby improving the efficiency of the model.

[0191] Step S23: Construct a graph reconstruction module based on geometry and label information (GSRM-GL), which makes up for the deficiency of the initial graph in measuring the relationship between superpixel nodes and further mines the deep spatial-spectral information hidden in the hyperspectral image.

[0192] 1. Initial construction of super-pixel regions. The Boruvka minimum spanning tree algorithm is used to divide the hyperspectral image into super-pixel regions of multiple scales. Each super-pixel region can be regarded as the result of an unsupervised clustering.

[0193] Each superpixel region is regarded as a collection of similar pixels for subsequent graph structure reconstruction.

[0194] 2. Calculate label information:

[0195] 2.1 Using the feature representation H of the model output fuse and feature matrices at different levels Generate multiple cluster centers.

[0196] 2.2 For each superpixel region, calculate its cluster center in spatial and spectral dimensions to form feature representations at different levels.

[0197] 2.3 Feature H fuse Divide into labeled sets and unlabeled sets Two parts: Calculate the center of the labeled set for subsequent geometric similarity calculation:

[0198]

[0199] Among them C n Indicates the label node center of the nth category, y v represents the true label of the labeled node, J represents the total number of categories, and M(·) represents the mean function.

[0200] 3. Calculate geometric distance

[0201] 3.1 Use cosine distance to measure the directional difference between superpixel centers. The formula is as follows:

[0202]

[0203] in, and are the feature vectors of the superpixel region and label center, respectively.

[0204] 3.2 Use the normalized Euclidean distance to measure the spatial difference of superpixel centers. First, normalize the features and then calculate the normalized Euclidean distance:

[0205]

[0206] 3.3 Combine cosine distance and normalized Euclidean distance to calculate fusion similarity:

[0207]

[0208] Wherein, α1 and α2 are weight coefficients, satisfying α1+α2=1.

[0209] Step S24: Pseudo-label assignment

[0210] 4.1 For each superpixel region, select the label center with the minimum distance based on the fusion similarity and assign a pseudo label:

[0211]

[0212] Represents the i-th superpixel area in the l-th layer and the center of the nth class The geometric similarity metric is to find the best matching feature between the category center and the current superpixel region by selecting the minimum distance, and assign a pseudo label to each superpixel region in the lth layer.

[0213] 5. Redefinition of Graph Structure

[0214] The adjacency relationship of the graph structure is redefined by combining geometric similarity and label information. The new adjacency matrix is ​​defined as follows:

[0215]

[0216] Among them, spatial similarity and label information jointly determine the connectivity between nodes, ensuring that the graph structure only retains nodes with high similarity.

[0217] The newly constructed graph structure is used to retrain the classification model to further improve the classification accuracy.

[0218]

[0219] in

[0220] Step S3, model training: train the model constructed in step S2 until the entire model converges and the optimal model is retained.

[0221] 1. Experimental Environment

[0222] To more scientifically evaluate the effectiveness of the proposed method, several representative algorithms were selected for comparison. These included three methods based on convolutional neural networks: deep convolutional neural networks (CNNs), spatial-spectral residual networks (SSRNs), and 2D-3D hybrid spectral networks (HybridSNs). Four methods based on a combination of convolutional neural networks and graph convolutional networks: CEGCN, WFCG, MSSGU, and AMGCF. The hyperparameters obtained for the compared methods were consistent with those reported in the original literature. Each value in the table uses the mean average precision and associated standard deviation of ten experimental results to ensure stable and reliable results on each HSI dataset. The experiments were run on a computer equipped with an NVIDIA GeForce RTX 4060Ti (16GB of GPU memory) and an i5-12600KF CPU. Python 3.9 was used as the programming language, PyTorch 2.4.0 was selected as the programming environment, and CUDA version 12.1 was used.

[0223] 2. Define hyperparameters:

[0224] The learning rate is set to 5e-4. If the number of training iterations reaches the number of training iterations Epoch (300 times), the training is terminated to obtain the trained hyperspectral image.

[0225] 3. The spatial-spectral joint loss function is forward propagated and then back-propagated using the Adam optimizer to optimize all parameters of the hyperspectral image classification neural network.

[0226] Use the spatial-spectral joint loss function L total The overall expression is:

[0227] L total =L CE +λ spatial ·L TV,spatial +λ spectral ·L TV,spectral

[0228] Where: L CE is the cross entropy loss, which is used to measure the difference between the model prediction category and the true category; L TV,spatial is the total spatial variation loss, which is used to maintain the smoothness of the image space; L TV,spectral is the total spectral variation loss, which ensures the smoothness of the image in the spectral dimension; spatial and λ spectral It is a weight parameter used to adjust the contribution of different loss items to the total loss to balance the classification accuracy and the coherence of spatial spectral features.

[0229] Step S4, model validation reasoning: input the test set data into the trained optimal model, infer the pixel category prediction value of the test image, and evaluate the prediction index.

[0230] The specific process of step S4 is as follows: after completing the model training, the model parameters are fixed, the hyperspectral images and labels in the test set are input into the optimal model, the category to which each pixel belongs is predicted, and the overall accuracy OA, average accuracy AA and Kappa coefficient between the true category and the predicted category are calculated to evaluate the effectiveness of the classification method.

[0231] Overall accuracy OA:

[0232]

[0233] Average Precision (AA):

[0234]

[0235] Kappa coefficient:

[0236]

[0237] Assume that the number of real samples of each class is a1,…ac, and the number of samples of each class predicted is b1,b2…,bc, the total number of samples is b1,b2,…bc, and the total number of samples is ρ, then

[0238] like Figure 5 As shown, the input Salinas original hyperspectral image and its true value, the visual classification results of the Salinas dataset (a) ground truth map (b) false color image (c) CNN (OA = 69.66%) (d) SSRN (OA = 86.92%) (e) HybridSN (OA = 90.73%) (f) CEGCN (OA = 92.63%) (g) WFCG (OA = 92.58%) (h) MSSGU (OA = 93.93%) (i) AMGCF (OA = 92.88%) (j) Proposed (FMGDR) (OA = 97.77%), by Figure 5 It can be seen that the model recognition accuracy of the present invention is higher. Thus, the operation of model testing and reasoning is completed.

[0239] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction, characterized by: The following steps are involved: S1, obtaining the original hyperspectral image data and preprocessing it, reducing redundant information through spectral dimension reduction method, and obtaining the hyperspectral data after dimension reduction; S2. Construct a multi-scale GCN and DCN hyperspectral image classification neural network based on graph reconstruction. The network includes a dynamic spectral feature extraction module DSFEM and a multi-scale spatial-spectral graph convolution module MSSGCM. S3, the graph reconstruction module GSRM-GL builds a new graph structure based on geometric information and label information to enhance the model's understanding of spatial and spectral information; S4. Use training samples of known categories to train the network to obtain a trained classification model. S5. Apply the spatial-spectral joint loss function to further optimize the classification model, including cross entropy loss, spatial total variation loss, and spectral total variation loss, to ensure classification accuracy, spatial information smoothness, and spectral information continuity; S6. Input the hyperspectral image data to be classified into the trained model, use the classification neural network to perform pixel-level classification on it, and finally output the classification result of each pixel.

2. The multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction according to claim 1 is characterized in that In the dynamic spectral feature extraction module DSFEM, 1x1 convolution and deformable convolution DCN are used to extract the spectral-spatial features of the local irregular area. The deformable convolution DCN is implemented by the following steps: a. Generate an initial set of sampling point coordinates and learn offsets through a trainable convolutional layer to adjust the sampling positions; b. Use bilinear interpolation to calculate the eigenvalues ​​of the new sampling points and extract the vertical spectral features through column convolution; c. Batch normalization and nonlinear activation function SiLU are used to process output features.

3. The multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction according to claim 1 is characterized in that The multi-scale spatial-spectral graph convolution module (MSSGCM) extracts global features at different scales in parallel. Specifically, it includes the following steps: a. Use the Boruvka algorithm to divide the dimensionality-reduced Salinas image into five layers of superpixel structures with different segmentation scales through progressive merging. The number of superpixels decreases as the scale increases. b. Construct the Salinas superpixel relationship matrix and construct the convolution input features for the multi-scale GCN module graph; c. Construct an adjacency matrix and define the graph structure based on the definition of spatial similarity and the selection of neighbor nodes; d. Perform multi-scale graph convolution calculation and multi-scale feature integration; e. Input Salinas pixel-level features and Salinas multi-scale superpixel-level features, and concatenate them along the channel dimension to generate a unified feature representation; f. Use depth-wise separable convolution (SSConv) to perform convolution operations on the fused features to generate further fused features.

4. The multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction according to claim 1 is characterized in that The graph reconstruction module GSRM-GL is implemented by the following steps: a. Calculate the geometric center of the label category and measure the similarity between the superpixel and the label center based on cosine distance and normalized Euclidean distance; b. Assign pseudo labels to superpixels and reconstruct the adjacency matrix to preserve high similarity node connections; c. Use the reconstructed graph structure to retrain the model and improve classification accuracy.

5. The multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction according to claim 1 is characterized in that The spatial-spectral joint loss function L total The expression is: L total =L CE +λ spatial ·L TV,spatial +λ spectral ·L TV,spectral Where: L CE is the cross entropy loss, which is used to measure the difference between the model prediction category and the true category; L TV,spatial is the total spatial variation loss, which is used to maintain the smoothness of the image space; L TV,spectral is the total spectral variation loss, which ensures the smoothness of the image in the spectral dimension; spatial and λ spectral It is a weight parameter used to adjust the contribution of different loss items to the total loss to balance the classification accuracy and the coherence of spatial spectral features.

6. The multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction according to claim 1 is characterized in that The classification result evaluation indicators include: a. Overall accuracy OA: the proportion of correctly classified samples to the total samples; b. Average accuracy AA: the average value of classification accuracy of each category; c.Kappa coefficient: measures the consistency between the classification results and random predictions.

7. The multi-scale GCN and DCN hyperspectral image classification method based on graph reconstruction according to claim 2 is characterized in that The calculation formula of the bilinear interpolation is: x offset =g lt X(q lt )+g rb X(q rb )+g lb X(q lb )+g rt X(q rt ) g lt =(1+(q lt,x -p x ′))·(1+(q lt,y -p′ y )) g rb =(1-(q rb,x -p x ′))·(1-(q rb,y -p′ y )) g lb =(1+(q lb,x -p x ′))·(1-(q lb,y -p′ y )) g rt =(1-(q rt,x -p x ′))·(1+(q rt,y -p′ y )) Among them, q lt ,q rb ,q lb ,q rt Represents the coordinates of the four integer sampling points at the upper left, lower right, lower left and upper right corners respectively; X(q lt ),X(q rb ),X(q lb ),X(q rt ) represent the eigenvalues ​​of these four sampling points in the input feature map; g lt ,g rb ,g lb ,g rt is the interpolation weight.

Images