Hyperspectral image classification method of graph self-attention network based on greedy strategy guidance

Abstract

The invention relates to a hyperspectral image classification method, in particular to a hyperspectral image classification method based on a greedy strategy guided graph self-attention network. The invention provides a hyperspectral image classification method of a graph self-attention network based on greedy strategy guidance in order to solve the problems that in existing HSI classification, samples are limited, GAT tends to only consider directly-connected nodes, indirectly-related relations are insufficiently considered, and the classification effect is affected. The method comprises the following steps: step 1, constructing a graph self-attention network based on greedy strategy guidance; the greedy strategy guidance-based graph self-attention network comprises a data preprocessing stage, a super-pixel-based graph self-attention branch stage, a pixel-based multi-attention fusion branch stage, a classification stage and a greedy training strategy stage. And step 2, classifying the hyperspectral image to be measured based on the trained greedy strategy guided graph self-attention network to obtain a classification result. The method is applied to the field of hyperspectral image classification.

Description

Technical Field

[0001] The invention relates to a hyperspectral image classification method. Background Art

[0002] Hyperspectral images (HSI) integrate imaging technology and spectral detection technology, so that each pixel not only carries the spatial characteristics of the target, but also contains dozens to hundreds of continuous and subdivided spectral information, thereby achieving a detailed depiction of the surface cover [1] (P. Ghamisi et al., "Advances in Hyperspectral Image and Signal Processing: A Comprehensive Overview of the State of the Art," in IEEE Geoscience and Remote Sensing Magazine, vol. 5, no. 4, 2017, pp. 37-78.). HSI is also widely used in medical diagnosis due to its "image-atlas integration" feature[2][3]([2]G.Zhan,Y.Uwamoto andY.-W.Chen,"HyperUNet for Medical Hyperspectral Image Segmentation on a Choledochal Database,"2022IEEE International Conference on Consumer Electronics(ICCE),Las Vegas,NV,USA,2022,pp.1-5.[3]Y.Li,R.Wu,Q.Tan,Z.Yang and H.Huan,"Masked Spectral Bands Modeling With Shifted Windows:AnExcellent Self-Supervised Learner for Classification of Medical Hyperspectral Images,"in IEEE Signal Processing Letters,vol.30,2023,pp.543-547.), mineral exploration[4](L.Zhang,M.Zhang,J.Huang,C.Zhang,F.Ye and W.Pan,"A New Approach forMineral Mapping Using Drill-Core Hyperspectral Image," in IEEE Geoscience andRemote Sensing Letters, vol.20, 2023, pp.1-5.), environmental monitoring [5][6][7]([5]L.Chen,K.Tan, X., Wang, and C. Pan, "Estimation Soil Organic Matter Using Airborne Hyperspectral Imagery," 2023 13th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), Athens, Greece, 2023, pp. 1-5. [6] N.P, M.K.C.V.S.S, D.S.Pankaj, and R.R.Nidamanuri, "Solid Waste Detection and Waste-material Characterization in Urban Environment at Subpixel Level in Airborne Hyperspectral Imagery," 2023 13th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), Athens, Greece, 2023, pp. 1-3. [7] A.B.R, M.K.C.V.S.S, D.S.Pankaj, and R.R.Nidamanuri, "Invasive Plant Species Detection in Airborne Hyperspectral Imagery over Complex Forest Landscape," 2023 13th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), Athens, Greece, 2023, pp. 1-3.) Military reconnaissance [8] (M. Shimoni, R. Haelterman, and C.Perneel,"Hypersectral Imaging for Military and Security Applications:Combining Myriad Processing and Sensing Techniques,"inIEEE Geoscience and Remote Sensing Magazine,vol.7,no.2,2019,pp.101-117.) and other fields. Therefore, the research on hyperspectral image classification methods has also attracted much attention in the field of remote sensing [9](S.Li,W.Song,L.Fang,Y.Chen,P.Ghamisi and JABenediktsson,"Deep Learning for Hyperspectral Image Classification:An Overview,"in IEEE Transactions on Geoscience and Remote Sensing,vol.57,no.9,2019,pp.6690-6709.). .

[0003] Early HSI classification methods mainly focused on extracting feature information from spectral data. Representative methods include logistic regression

[10] (R. Elkadiri et al., "A Remote Sensing-Based Approach for Debris-Flow Susceptibility Assessment Using Artificial Neural Networks and Logistic Regression Modeling," in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 7, no. 12, Dec. 2014, pp. 4818-4835.), random forest

[11] (MHRSales, S. de Bruin, C. Souza and M. Herold, "Land Use and Land Cover Area Estimates From Class Membership Probability of a Random Forest Classification," in IEEE Transactions on Geoscience and Remote Sensing, vol. 7, no. 12, Dec. 2014, pp. 4818-4835.), and random forest

[11] (MHRSales, S. de Bruin, C. Souza and M. Herold, "Land Use and Land Cover Area Estimates From Class Membership Probability of a Random Forest Classification," in IEEE Transactions on Geoscience and Remote Sensing, vol. 7, no. 12, Dec. 2014, pp. 4818-4835.). Sensing, vol.60, 2022, pp.1-11.), Support Vector Machine

[12]

[13]

[14] (

[12] M.Sheykhmousa,M.Mahdianpari,H.Ghanbari,F.Mohammadimanesh,P.Ghamisi and S.Homayouni,"Support VectorMachine Versus Random Forest for Remote Sensing Image Classification:AMeta-Analysis and Systematic Review," in IEEE Journal of Selected Topics in AppliedEarth Observations and Remote Sensing, vol.13, 2020, pp.6308-6325.

[13] H.Li, "An Overview on Remote Sensing Image Classification Methods with a Focus on Support Vector Machine," 2021 International Conference on Signal Processing and Machine Learning (CONF-SPML), Stanford, CA, USA, 2021, pp. 50-56.

[14] J.L. Garrett, N.K. Singh, T.A. Johansen and I. Necoara, "Accelerating Support Vector Machines For Remote Platforms By Increasing Sparsity," 2022 12th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), Rome, Italy, 2022, pp. 1-5.) and sparse representation classification

[15]

[16] (

[15] J. Peng et al., "Low-Rank and Sparse Representation for Hyperspectral Image Processing: A review," in IEEE Geoscience and Remote Sensing Magazine, vol. 10, no. 1, 2022, pp. 10-43.

[16] L. Zhuang, L. Gao, B. Zhang, X. Fu and J.M. Bioucas-Dias, "Hyperspectral Image Denoising and Anomaly Detection Based on Low-Rank and Sparse Representations," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, 2022, pp.1-17), etc. However, these methods have certain limitations in application, mainly reflected in the insufficient exploration of spatial dimensional features and over-reliance on the prior knowledge of experts. In recent years, deep learning has made great breakthroughs in the fields of computer vision and natural language processing, which has also attracted the attention of researchers in the field of HSI classification

[17]

[18]

[19] (

[17] Q.Wang,B.Wu,P.Zhu,P.Li,W.Zuo and Q.Hu,"ECA-Net:Efficient ChannelAttention for Deep Convolutional Neural Networks,"2020IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR),Seattle,WA,USA,2020,pp.11531-11539.

[18] J.Wang et al.,"Deep High-Resolution Representation Learning forVisual Recognition,"in IEEE Transactions on Pattern Analysis and Machine Intelligence,vol.43,no.10,2021,pp.3349-3364.

[19] T.Young,D.Hazarika,S.Poriaand E.Cambria,"Recent Trends in Deep Learning Based Natural Language Processing [Review Article]," in IEEE Computational Intelligence Magazine, vol. 13, 2018, no. 3, pp. 55-75.). Deep learning has a powerful feature learning ability, which can automatically capture and extract deeper abstract and unique features, thus getting rid of the dependence on manual feature labeling and prior knowledge. Among them, convolutional neural network (CNN)

[20] (Y. Chen, H. Jiang, C. Li, X. Jia and P.Ghamisi, "Deep Feature Extraction and Classification ofHyperspectralImages Based on Convolutional Neural Networks," in IEEE Transactions onGeoscience and Remote Sensing, vol.54, no.10,2016, pp.6232-6251.), Transformer

[21] (Vaswani, Ashish, et al. "Attention is all you need," Advances in neuralinformation processing systems 30, 2017.) and graph neural network (GNN)

[22]

[23] (

[22] Z. Zhang, P. Cui and W. Zhu, "Deep Learning on Graphs: A Survey," in IEEE Transactions on Knowledge and Data Engineering, vol.34, no.1, 2022, pp.249-270.

[23] D.Hong, L.Gao, J.Yao, B.Zhang, A.Plaza and J.Chanussot, "GraphConvolutional Networks for Hyperspectral Image Classification," in IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 7, 2021, pp. 5966-5978.) has achieved the most remarkable results. The 3D convolution kernel in CNN has shown excellent adaptability in the feature extraction of HSI. Hyperspectral images contain rich spectral information and continuous spatial distribution of objects. The 3D convolution kernel can simultaneously consider the spatial information and spectral information of the image, thereby extracting the features of HSI more comprehensively. An end-to-end spectral-spatial residual network (SSRN) was proposed

[24] (Z. Zhong, J. Li, Z. Luo and M.Chapman, "Spectral–Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework," in IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 2, 2018, pp. 847-858.). This method combines 3D convolution kernels with ResNet

[25] (K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 770-778.), which not only extracts spectral and spatial features at the same time, but also further enhances the abstraction and expression capabilities of features by deepening the number of network layers. However, 3D convolution kernels also have the problem of low computational efficiency; and compared with traditional 2D convolution kernels, 3D convolution kernels involve more computation and require more computing resources. Therefore, a hybrid convolutional network (hybridspectral CNN, HybridSN)

[26] (SK Roy, G. Krishna, SR Dubey and BBChaudhuri, "HybridSN: Exploring 3-D–2-D CNN Feature Hierarchy for Hyperspectral Image Classification," in IEEE Geoscience and Remote Sensing Letters, vol. 17, no. 2, 2020, pp. 277-281.) was proposed for high-resolution remote sensing image classification tasks. This method combines 3D convolution kernels and 2D convolution kernels to fully utilize their spatial and spectral feature representation capabilities. Although these two methods have constructed quite sophisticated network structures, it is difficult to further improve the effect of feature extraction using only convolution. .

[0004] Attention mechanism is an important technology in deep learning. Its core idea is to enable the model to dynamically focus on certain parts of the input data when processing information

[27]

[28]

[29] (

[27] J.Hu,L.Shen and G.Sun,"Squeeze-and-Excitation Networks,"2018IEEE / CVF Conference on Computer Visionand Pattern Recognition,Salt Lake City,UT,USA,2018,pp.7132-7141.

[28] J.Fu,J.Liu,J.Jiang,Y.Li,Y.Bao and H.Lu,"Scene Segmentation With Dual Relation-Aware Attention Network,"in IEEE Transactions on Neural Networks and Learning Systems,vol.32,no.6,2021,pp.2547-2560.

[29] S.Ren,K.He,R.Girshick and J.Sun,"Faster R-CNN:Towards Real-Time Object Detection with Region Proposal Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, 2017, pp. 1137-1149.). This mechanism mimics the attention allocation in human vision and cognition, allowing the model to focus more computing resources on the most relevant information according to the needs of the task. A dual-triple attention network (DTAN)

[30] (Y. Cui, Z. Yu, J. Han, S. Gao and L. Wang, "Dual-Triple Attention Network for Hyperspectral Image Classification Using Limited Training Samples," in IEEE Geoscience and Remote Sensing Letters, vol. 19, 2022, pp. 1-5) for hyperspectral image classification with limited training samples was proposed. The model uses a dual attention mechanism to capture the interaction between spatial information and spectral information.A multi-scale spectral-spatial attention network (MOCNN)

[31] (Liang, Lianhui, Shaoquan Zhang, Jun Li, Antonio Plaza, and Zhi Cui. "Multi-Scale Spectral-Spatial Attention Network for Hyperspectral Image Classification Combining 2D Octave and 3D Convolutional Neural Networks" Remote Sensing vol.15, no.7, 2023.) was proposed. This method uses multi-scale 2D octave convolution and 3D DenseNet

[32] (G. Huang, Z. Liu, L. Van Der Maaten and KQ Weinberger, "Densely Connected Convolutional Networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp.2261-2269.) to extract spatial and spectral features. In addition, two attention mechanism modules are used to enhance network performance. A method based on adaptive subspaces classifier and feature transformation (SSFT)

[33] (J.Bai et al., "Few-Shot Hyperspectral Image Classification Based on Adaptive Subspaces and Feature Transformation," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, 2022, pp. 1-17.) is proposed. This method uses a 3D local channel attention and feature transformation module to alleviate the problem of limited samples.An automatic attention-based convolutional neural network (AAtt-CNN)

[34] (MEPaoletti,S. Y.Xue, JM Haot and A.Plaza,"AAtt-CNN:AutomaticAttention-Based Convolutional Neural Networks for Hyperspectral ImageClassification,"in IEEE Transactions on Geoscience and Remote Sensing,vol.61,2023,pp.1-18) was proposed. This method uses the channel attention mechanism to extract the most relevant features while reducing the amount of data required for training.

[0005] Although the method of combining attention mechanism with convolution kernel has achieved remarkable results in the field of HSI classification, the problem of limited receptive field of convolution kernel has always been an obstacle to further improving classification performance. The size of the receptive field directly determines the range of image context information that the convolution kernel can capture. When processing complex data such as hyperspectral images, a wider receptive field is often required to capture richer features. Transformer was first proposed in the field of natural language processing (NLP) and quickly became a revolutionary architecture in this field, especially in machine translation tasks. It has achieved unprecedented results. Subsequently, researchers began to explore the transfer of the powerful capabilities of the Transformer model to other fields. After a series of improvements and adaptations, the Transformer model gradually demonstrated its excellent performance. Based on the characteristics of image data, researchers have designed a series of Transformer variants suitable for visual tasks, such as Vision Transformer, Swin-Transformer, etc.

[35]

[36]

[37]

[38]

[39]

[40] (

[35] A.Dosovitskiy,L.Beyer,A.Kolesnikov,D.Weissenborn,X.Zhai,T.Unterthiner,M.Dehghani,M.Minderer,G.Heigold,S.Gelly,J.Uszkoreit,and N.Houlsby,“An image is worth 16x16 words:Transformers forimage recognition at scale,”2020.

[36] Daquan Zhoul,Bingyi Kang,Xiaojie Jin,Linjie Yang,Xiaochen Lian,Qibin Hou,et al.,“DeepViT:Towards Deeper Vision Transformer”,arXiv,Mar2021.

[37] H.Wu et al.,“DeepViT:Towards Deeper Vision Transformer”,arXiv,Mar2021. al., "CvT: Introducing Convolutions to Vision Transformers," in 2021 IEEE / CVF International Conference on ComputerVision (ICCV), Montreal, QC, Canada, Oct. 2021.

[38] L. Yuan et al., "Tokens-to-tokenViT: Training vision transformers from scratch on ImageNet", Proc.IEEE / CVFInt.Conf.Comput.Vis.,pp.558-567,2021.

[39] Z.Liu et al.,"Swin Transformer:Hierarchical Vision Transformer using Shifted Windows,"2021IEEE / CVFInternational Conference on Computer Vision(ICCV),Montreal,QC,Canada,2021,pp.9992-10002.

[40] W.Wang et al.,"Pyramid Vision Transformer:AVersatileBackbone for Dense Prediction without Convolutions,"2021IEEE / CVFInternational Conference on Computer Vision(ICCV),Montreal,QC,Canada,2021,pp.548-558.). These models effectively capture the global and local features in the image by dividing the image into multiple small patches and feeding these patches as sequences into the Transformer for processing. A Transformer-based HSI classification method SpectralFormer

[41] (D.Hong et al., "SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers," in IEEE Transactions on Geoscience and Remote Sensing, vol.60, 2022, pp.1-15.) was proposed. It captures local details by learning the relationship between adjacent spectra within a group and uses cross-layer skip connections to transfer memory information. A spectral-spatial feature tokenization transformer (SSFTT)

[42] (L.Sun, G.Zhao, Y.Zheng and Z.Wu,"Spectral–SpatialFeature Tokenization Transformer for Hyperspectral Image Classification,"inIEEE Transactions on Geoscience and Remote Sensing,vol.60,2022,pp.1-14.) was proposed. This method uses convolutional layers to extract low-level features, then converts them into semantic tags and models high-level semantic features through Transformer. A dual branch method based on multi-scale feature extraction and Transformer network (dual branchmultiscale transformer network,DBMST)

[43] (C.Shi,S.Yue and L.Wang,"A Dual-Branch Multiscale Transformer Network for Hyperspectral ImageClassification,"in IEEE Transactions on Geoscience and Remote Sensing,vol.62,2024,pp.1-20.) was proposed. This method achieves better classification results by extracting local features using convolutional neural networks and fusing them with global features. .

[0006] The application of GNN in the field of hyperspectral image classification has also been a research hotspot in recent years. The application of GNN in hyperspectral image classification is mainly to regard the local area composed of each pixel and its neighboring pixels as a node in the graph, and the connection between nodes is defined according to the spatial relationship or spectral similarity between pixels. Through the GNN model, the rich spectral information and spatial information of hyperspectral data can be effectively integrated, thereby improving the accuracy of classification. A method based on the combination of convolutional neural network and graph convolutional network (CNN-enhanced graph convolutional network, CEGCN)

[44] (Q.Liu, L.Xiao, J.Yang and Z.Wei, "CNN-Enhanced Graph Convolutional Network WithPixel-and Superpixel-Level Feature Fusion for Hyperspectral Image Classification," in IEEE Transactions on Geoscience and Remote Sensing, vol.59, no.10, 2021, pp.8657-8671.) was proposed. This method uses convolutional neural network to extract local pixel-level features, while graph convolutional network (GCN) is used to model the features of large-scale irregular areas. A fast parallel network based on dynamic graph convolutional network and convolutional neural network (FDGC)

[45] (Q.Liu, Y.Dong, Y.Zhang and H.Luo, "A Fast DynamicGraph Convolutional Network and CNN Parallel Network for Hyperspectral ImageClassification," in IEEE Transactions on Geoscience and Remote Sensing, vol.60, 2022, pp.1-15.) was proposed. This method combines dynamic graph convolutional network and convolutional neural network, and uses three branches to extract different types of features in parallel.A method called weighted feature fusion of convolutional neural network and graph attention network (WFCG)

[46] (Y. Dong, Q. Liu, B. Du and L. Zhang, "Weighted Feature Fusion of Convolutional Neural Network and Graph Attention Network for Hyperspectral Image Classification," in IEEE Transactions on Image Processing, vol. 31, 2022, pp. 1559-1572.) was proposed. This method combines CNN and graph attention network (GAT) to give full play to their respective advantages. A method based on the fusion of convolutional and enhanced graph attention network (CNN and enhanced-GAT, CEGAT)

[47] (C. Shi, H. Wu and L. Wang, "CEGAT: A CNN and enhanced-GAT based on key sample selection strategy for hyperspectral image classification," Neural Networks, Vol. 168, 2023, pp. 105-122.) was proposed. This method is used to alleviate the problem of hyperspectral image classification under limited labeled samples. A method based on multi-scale convolution and multi-hop graph convolution fusion (attention multihop graph and multiscale convolutional fusion network, AMGCFN)

[48] (H. Zhou, F. Luo, H. Zhuang, Z. Weng, X. Gong and Z. Lin, "Attention Multihop Graph and Multiscale Convolutional Fusion Network for Hyperspectral Image Classification," in IEEE Transactions on Geoscience and Remote Sensing, vol. 61, 2023, pp. 1-14.) was proposed.This method is used to extract multi-level complex features in HSI classification.

[0007] Although the above methods have achieved good classification performance, there are still some problems. On the one hand, sample limitation is a long-standing problem in HSI classification. Hyperspectral images contain a large number of spectral bands, which makes the spectral dimension high, and thus requires a large number of labeled samples to train the classifier. However, the cost of labeling samples is high, and the lack of samples may cause model overfitting. On the other hand, GAT strengthens the understanding of graph structure and the capture of connections between nodes through the attention mechanism, but it tends to only consider directly connected nodes, and does not consider indirectly related connections enough. For HSI classification, this will affect the capture of potentially related information between nodes, thereby affecting the classification effect. At the same time, GAT pays less attention to the intrinsic properties of the nodes themselves, which further limits its application potential. Summary of the invention

[0008] The purpose of this invention is to solve the problems of limited samples in existing HSI classification and GAT tending to only consider directly connected nodes but insufficient consideration of indirectly related connections, which affects the classification effect, and propose a hyperspectral image classification method based on a graph self-attention network guided by a greedy strategy.

[0009] The specific process of the hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy is as follows:

[0010] Step 1: Construct a graph self-attention network guided by a greedy strategy;

[0011] The graph self-attention network guided by the greedy strategy includes a data preprocessing stage, a superpixel-based graph self-attention branch stage, a pixel-based multi-attention fusion branch stage, a classification stage, and a greedy training strategy stage.

[0012] The data preprocessing stage consists of two parts:

[0013] 1) Preliminary preparation of hyperspectral image data:

[0014] Linear discriminant analysis LDA is used to extract the mid-band of hyperspectral image X;

[0015] The extracted bands are segmented into superpixels using the simple linear iterative clustering (SLIC) method to obtain the relationship matrix And the adjacency matrix

[0016] H and W represent the height and width of HSI respectively; N represents the number of graph nodes; represents real numbers;

[0017] The relationship matrix G is expressed as:

[0018] Among them, G i,j Represents the value of the relationship matrix at (i, j), i represents the i-th pixel, j represents the j-th superpixel; the relationship matrix G i,j The rows represent pixels and the columns represent superpixels; Flatten(·) represents the flattening operation. express The i-th pixel in S j represents the jth superpixel in S; represents a one-dimensional vector after flattening the spatial dimension; S represents a superpixel;

[0019] The adjacency matrix H is expressed as:

[0020] Among them, H i′,j′ represents the value of the adjacency matrix at (i′, j′), i′ represents the i′th superpixel, and j′ represents the j′th superpixel; the rows and columns of the adjacency matrix H represent superpixels, and the elements in the adjacency matrix H represent whether there is a connection between nodes (i′, j′); S i′ represents the i′th superpixel in S, S j′ represents the j′th superpixel in S;

[0021] 2) The hyperspectral image X is subjected to two point-by-point convolution operations to obtain the output features It is expressed as:

[0022] X cnn =PW(PW(X))

[0023] Where PW(·) represents a point-wise convolution block, which includes a batch normalization layer, a convolution layer, and a ReLU activation function layer in sequence; L represents the length of the spectral sequence after two point-wise convolution operations;

[0024] The specific working process of the graph self-attention network guided by the greedy strategy is as follows:

[0025] The hyperspectral image is input into the data preprocessing stage. The output image of the data preprocessing stage is input into the superpixel-based image self-attention branch stage and the pixel-based multi-attention fusion branch stage respectively. The features output by the superpixel-based image self-attention branch stage and the pixel-based multi-attention fusion branch stage are fused. The fused features are input into the classification stage, and the classification stage outputs the classification results.

[0026] Step 2: Classify the hyperspectral image to be tested based on the graph self-attention network guided by the greedy strategy to obtain the classification result.

[0027] The beneficial effects of the present invention are:

[0028] In the study of hyperspectral image classification (HSIC), sample limitation is a common problem. Due to the high dimensionality and large data volume of hyperspectral images, it is often very difficult and costly to obtain labeled samples, which limits the number of samples. Insufficient sample numbers will lead to insufficient training of the classifier, affecting the accuracy and stability of the classifier. To alleviate the above problems, a greedy strategy guided Graph self-attention network (GS-GraphSAT) is proposed. First, in order to enhance the information transfer between nodes and explore the sequence properties of the nodes themselves, a graph self-attention mechanism (GSA) is designed. GSA effectively combines the advantages of multi-head self-attention (MHSA) and graph attention network GAT, not only fully capturing the relationship between directly connected nodes, but also deeply exploring the connection between non-directly connected nodes with potential correlation. At the same time, with the help of the characteristics of MHSA, the intrinsic characteristics of the nodes are fully mined, further improving the representation ability of graph data. . Secondly, in order to fully obtain the local features at the pixel level, a multi-attention fusion module (MAF) is constructed. MAF extracts the local features of the image by applying convolution kernels of different sizes at multiple scales and fuses the weighted features of multiple attention mechanisms, thereby more comprehensively capturing and enhancing the local feature information of the image at the pixel level. Its design purpose is to deeply capture the local feature details of the image at the pixel level, thereby enriching and expanding the level and range of the features extracted by the network. Finally, in order to alleviate the problem of sample limitation in hyperspectral images, a greedy training strategy (GTS) is proposed. During the training process, GTS accurately judges the best time to supplement samples by analyzing the changes in loss, so that the classification accuracy of the model can be effectively improved in the case of limited samples. The proposed GS-GraphSAT is fully experimented on three challenging datasets. The experimental results show that compared with some current mainstream methods, the method proposed in this paper has better classification performance.

[0029] A graph self-attention mechanism is designed. GSA combines the advantages of multi-head self-attention mechanism and graph attention network. It not only considers the direct relationship between nodes, but also deeply explores the relationship between potential non-directly connected nodes, and enhances the representation ability of graph data by mining the intrinsic characteristics of nodes.

[0030] A multi-attention fusion module was built. MAF enhances the ability to extract local features of images at the pixel level by combining convolution kernels of multiple scales and attention mechanisms. By capturing more local feature information of the image, the depth and breadth of the network features are enhanced.

[0031] A greedy training strategy was proposed. GTS optimizes the sample utilization method and determines the timing of supplementing samples by monitoring the loss change to achieve more efficient sample training with the same sample size.

[0032] A hybrid feature extraction framework integrating graph self-attention network, multi-attention fusion module and greedy training strategy is constructed. Extensive experiments are conducted on three datasets, and the experimental results demonstrate that the proposed method can provide excellent classification performance for hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the overall structure diagram of GS-GRAPHSAT of the present invention; Figure 2 The structure of the GSA of the present invention (Activate means activating the attention graph using the adjacency matrix, Accumulate means accumulating the attention graphs of multiple heads; Figure 3 It is the structural diagram of the MAF of the present invention; Figure 4 It is a structural diagram of the Channel Attention of the present invention; Figure 5 It is a structural diagram of SE of the present invention; Figure 6 Sample distribution diagram of IndianPines dataset, (a)-(d): pseudo color map, training set, validation set and test set; Figure 7 Sample distribution diagram on the Pavia University dataset, (a)-(d): pseudo-color map, training set, validation set and test set; Figure 8 Sample distribution diagram on the WHU-Hi-LongKou dataset, (a)-(d): pseudo-color image, training set, validation set and test set; Fig. 9 Comparison chart of the number of consecutive loss increases required to trigger GTS for different data sets;

[0034] Fig.10 Classification diagrams obtained by different methods on the IP dataset; Fig.11 Classification diagrams obtained by different methods on the UP dataset; Fig.12 Classification diagrams obtained by different methods on the LK dataset; Fig.13 The confusion matrix diagrams obtained on the IP dataset, (a)-(d) are: DTAN, SSFTT, WFCG, GS-GraphSAT; Fig.14Heat maps of different modules on the UP dataset, (a)-(d) are: initial CNN, MAF, GSA, GSA+MAF, respectively. DETAILED DESCRIPTION

[0035] Specific implementation method 1: The specific process of the hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy in this implementation method is as follows:

[0036] Step 1: Construct a greedy training strategy guided Graph self-attention network (GS-GraphSAT);

[0037] The graph self-attention network guided by the greedy strategy includes a data preprocessing stage, a superpixel-based graph self-attention branch stage, a pixel-based multi-attention fusion branch stage, a classification stage, and a greedy training strategy stage.

[0038] The data preprocessing stage consists of two parts:

[0039] 1) Preliminary preparation of hyperspectral image data:

[0040] Linear discriminant analysis (LDA)

[49] (Q.Ye, J.Yang, F.Liu, C.Zhao, N.Ye and T.Yin, "L1-Norm Distance Linear Discriminant AnalysisBased on an Effective Iterative Algorithm," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 1, 2018, pp. 114-129.) is used to extract the most discriminative bands in the hyperspectral image X to reduce the subsequent computational cost;

[0041] The extracted bands are segmented into superpixels using the simple linear iterative clustering (SLIC) method to obtain the relationship matrix And the adjacency matrix

[0042] H and W represent the height and width of HSI respectively; N represents the number of graph nodes (superpixels); represents real numbers;

[0043] The extracted bands (data after dimensionality reduction) are segmented into superpixels using the simple linear iterative clustering (SLIC) method

[50] (Y.-J. Liu, C.-C. Yu, M.-J. Yu and Y. He, "Manifold SLIC: A Fast Method to Compute Content-Sensitive Superpixels," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 651-659); adjacent pixels of the same type will be divided into the same superpixel block, so the relationship matrix can be obtained And the adjacency matrix

[0044] The relationship matrix G is expressed as:

[0045] Among them, G i,j Represents the value of the relationship matrix at (i, j), i represents the i-th pixel, j represents the j-th superpixel, reflecting the mapping relationship between pixels and superpixels, the relationship matrix G i,j The rows represent pixels and the columns represent superpixels; Flatten(·) represents the flattening operation. express The i-th pixel in S j represents the jth superpixel in S; represents a one-dimensional vector after flattening the spatial dimension; S represents a superpixel, which is used for Form a relationship matrix;

[0046] The adjacency matrix H is expressed as:

[0047] Among them, H i′,j′ represents the value of the adjacency matrix at (i′, j′), i′ represents the i′th superpixel, j′ represents the j′th superpixel, reflecting the connection relationship between superpixel nodes; the rows and columns of the adjacency matrix H represent superpixels, and the elements in the adjacency matrix H represent whether there is a connection between nodes (i′, j′); the adjacency matrix facilitates the rapid search for connection relationships between nodes; S i′ represents the i′th superpixel in S, S j′ represents the j′th superpixel in S;

[0048] 2) The hyperspectral image X is subjected to two point-by-point convolution operations to obtain the output features

[0049] First, the spectral dimension is transformed and point-by-point convolution is used to reduce the data dimension. Then, point-by-point convolution is used again for data cleaning to enhance the nonlinearity of the reduced-dimensional data. It can be expressed as:

[0050] X cnn =PW(PW(X))

[0051] Where PW(·) represents a point-wise convolution block, which includes a batch normalization layer, a convolution layer, and a ReLU activation function layer in sequence; L represents the length of the spectral sequence after two point-wise convolution operations;

[0052] The specific working process of the graph self-attention network guided by the greedy strategy is as follows:

[0053] The hyperspectral image is input into the data preprocessing stage. The output image of the data preprocessing stage is input into the superpixel-based image self-attention branch stage and the pixel-based multi-attention fusion branch stage respectively. The features output by the superpixel-based image self-attention branch stage and the pixel-based multi-attention fusion branch stage are fused. The fused features are input into the classification stage, and the classification stage outputs the classification results.

[0054] Step 2: Classify the hyperspectral image to be tested based on the graph self-attention network guided by the trained greedy strategy to obtain the classification result.

[0055] In order to cope with the challenge of limited samples in hyperspectral image classification, this paper proposes a graph self-attention network (GS-GraphSAT) guided by a greedy strategy. First, a graph self-attention mechanism (GSA) is constructed to deeply explore the feature connections between nodes at the superpixel level and the sequential properties of the nodes themselves. Secondly, a multi-attention fusion module (MAF) is designed, which can accurately learn the key spectral dimension features and spatial dimension distribution at the pixel level, thereby further improving the classification accuracy. In addition, a greedy training strategy (GTS) is designed to accurately judge the timing of supplementary samples by analyzing the loss changes during training, ensuring that the feature information in the samples is more efficiently learned and utilized. Finally, the extracted features at the superpixel level and pixel level are fused. Extensive experiments are carried out on IP, UP and LK datasets. The experimental results show that compared with the current mainstream classification methods, the proposed method GS-GraphSAT has more excellent classification performance. Looking forward to the future, multimodal remote sensing image classification has become an important trend in the development of this field, and we will also work towards this direction based on the research foundation of this paper.

[0056] Specific implementation method 2: The difference between this implementation method and specific implementation method 1 is that the superpixel-based graph self-attention branch stage in step 1 sequentially includes an encoding layer, a first graph self-attention mechanism GSA (graph self-attention mechanism), a second graph self-attention mechanism GSA (graph self-attention mechanism), and a decoding layer.

[0057] The other steps and parameters are the same as those in the first embodiment.

[0058] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that: the working process of the superpixel-based image self-attention branch stage in step one is:

[0059] 1) Output features of the data preprocessing stage and the relationship matrix G are input into the encoding layer, and the encoding layer outputs the transformed graph The process is expressed as:

[0060]

[0061] Among them, V represents the set of graph nodes composed of superpixels; represents the relationship matrix after column regularization of the relationship matrix G; the superscript T represents the transposition; Flatten(X cnn ) indicates flattening X; Encode(·) indicates encoding, which is used to convert the features output from the data preprocessing stage into Encoded into a set of nodes consisting of superpixels;

[0062] 2) Figure The adjacency matrix H output in the data preprocessing stage is input into the first graph self-attention mechanism GSA, and the first graph self-attention mechanism GSA outputs features; the first graph self-attention mechanism GSA output features and the adjacency matrix H output in the data preprocessing stage are input into the second graph self-attention mechanism GSA, and the second graph self-attention mechanism GSA outputs a weighted graph;

[0063] 3) Input the weighted graph and relationship matrix G into the decoding layer, and the decoding layer outputs pixel data X 1 ; The process is expressed as:

[0064] X 1 =Decode(V′;G)=Reshape(G×V′)

[0065] Among them, Decode(·) is used to decode node data into pixel data; Reshape(·) is used to restore the spatial dimension of flattened data; V′ represents the weighted graph node set output by the second graph self-attention mechanism GSA.

[0066] The other steps and parameters are the same as those in the first or second embodiment.

[0067] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that: the graph node in the 2) The adjacency matrix H output in the data preprocessing stage is input into the first graph self-attention mechanism GSA, and the first graph self-attention mechanism GSA outputs features; the specific working process is:

[0068] Graph attention network (GAT) is a method that applies attention mechanism on graph data, which can help the model understand the graph structure and capture the relationship between nodes. However, GAT also has some shortcomings; on the one hand, GAT mainly focuses on the relationship between directly connected nodes, which means that if there is no direct connection between nodes, the information transfer between them will be limited. For hyperspectral images, even if there is a potential similarity or correlation between two nodes, GAT cannot effectively capture this relationship, which in turn causes the model to be unable to fully utilize this information when classifying nodes; on the other hand, GAT lacks exploration of the properties of the nodes themselves. This means that the superpixel blocks processed by SLIC themselves cannot be fully utilized by GAT.

[0069] In order to alleviate the above problems, a graph self-attention mechanism (GSA) was proposed. Its detailed structure is as follows: Figure 2 shown.

[0070] (1) Use mapping operations to input graph data Transform into graph data as well as

[0071] Where L = h × d, h represents the number of multiple heads, and d represents the length of the sequence after multiple head segmentation;

[0072] right and Perform matrix multiplication to obtain the attention map M, and use the Softmax function to normalize the attention map M and combine it with Perform matrix multiplication to get the output result

[0073] The expression is:

[0074] The superscript T indicates transposition; d k express Dimensions;

[0075] (2) Output results of multiple attention heads The final result A is obtained by cascading;

[0076] (3) Accumulate the attention maps M of multiple heads to obtain the output M 1 ;

[0077] (4) Graph nodes Input graph attention network GAT (graph attention network), graph attention network GAT outputs attention map M 2 ;

[0078] (5) Attention map M 1 and attention map M 2 Perform element-by-element summation to obtain the attention map D;

[0079] (6) Activate the feature matrix D to obtain the activated feature matrix E. The process is as follows: retain the values ​​of the corresponding positions of the two nodes in the feature matrix D that have a connection relationship; set the values ​​of the corresponding positions of the two nodes that have no connection relationship in the feature matrix D to 0;

[0080] (7) Multiply element-by-element with the feature matrix E to obtain the weighted graph F;

[0081] (8) Add the weighted graph A and the weighted graph F element by element to obtain the summed graph J;

[0082] (9) Repeat (1) to (8) K times, concatenate the feature matrix results obtained K times, and use the concatenation result as the output feature of the first graph self-attention mechanism GSA.

[0083] Specifically, in the self-attention part, the mapping operation is first used to input the graph data into Transformed into as well as Three copies of graph data; where L = h × d, h represents the number of long heads, and d represents the length of the sequence after long-head segmentation;

[0084] right and Perform matrix multiplication to get the attention map Use the Softmax function to normalize the attention map M and Perform matrix multiplication to obtain multi-head output results Will output the result The final result of self-attention processing is obtained by merging multiple heads in It is expressed as:

[0085]

[0086] Among them, d k represent The dimension of , ||(·) represents the cascade operation, and W represents the parameter matrix;

[0087] In addition, attention maps of multiple attention heads are also required. Perform accumulation operation to obtain the attention map For use in the graph attention part; this process is expressed as:

[0088] Next, we will process the graph attention part; specifically, we first perform node feature Perform a linear transformation and use a shared attention mechanism to calculate the attention correlation coefficient; this process is expressed as:

[0089] Among them, e ij represents the attention correlation coefficient, which can represent the correlation of node i with respect to node j; W represents the parameter matrix, and A(·) represents the shared attention mechanism;

[0090] Afterwards, the Softmax function is used to normalize the attention correlation coefficient into a score. This process can be expressed as:

[0091]

[0092] Then, the attention map M after self-attention processing h With M g Perform matrix addition fusion;

[0093] Next, the adjacency matrix H is used to activate the fused result to obtain the final attention map

[0094] The specific process is:

[0095] For node positions with connection relationships, the fused results are retained; for node positions without connection relationships, the node values ​​are set to zero. This process can be expressed as:

[0096] After that, the activated result M f With the original graph input data Perform matrix multiplication to get the output of the graph attention part

[0097] M g On the basis of the original graph attention, the attention graph of multi-head self-attention is combined to enhance the interaction and feature expression of information between nodes, further improving the accuracy and robustness of the graph attention mechanism;

[0098] Finally, the output results of the two graph attention parts are and Fusion to get the final result This process can be expressed as:

[0099] In order to stably fuse the weighted feature maps of self-attention and graph attention, a multi-head mechanism is introduced in the first GSA layer, that is, the first GSA layer is executed K times independently, and then the results are cascaded; this process is expressed as:

[0100]

[0101] In the formula, σ r (·) represents the ReLU activation function;

[0102] A single-layer structure is adopted in the second GSA layer, and the final result is decoded by the Decode(·) function and then sent to the classifier.

[0103] By combining the advantages of MHSA and GAT, GSA not only accurately captures the relationship between directly connected nodes, but also deeply explores the relationship between potentially related non-directly connected nodes. At the same time, with the help of the characteristics of MHSA, the intrinsic characteristics of the nodes are also fully mined. Therefore, GSA can significantly improve the classification performance of the model.

[0104] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0105] Specific implementation mode 5: This implementation mode is different from any one of specific implementation modes 1 to 4 in that the value of K is 4.

[0106] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0107] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that: in the 2), the output features of the first graph self-attention mechanism GSA and the adjacency matrix H output in the data preprocessing stage are input into the second graph self-attention mechanism GSA, and the second graph self-attention mechanism GSA outputs a weighted graph; the specific process is:

[0108] 1) Use the mapping operation to output the graph data of the first graph self-attention mechanism GSA Transform into graph data as well as

[0109] right and Perform matrix multiplication to obtain the attention map M′, and use the Softmax function to normalize the attention map M′ and combine it with Perform matrix multiplication to get the output result

[0110] The expression is:

[0111] The superscript T means to find the transpose;

[0112] 2) Output results of multiple heads Cascade to obtain the feature matrix A′;

[0113] 3) Accumulate the attention graphs M′ of multiple heads to obtain the attention graph M′ 1 ;

[0114] 4) Figure Input graph attention network GAT (graph attention network), graph attention network GAT outputs attention map M′ 2 ;

[0115] 5) Attention map M′ 1 and attention map M′ 2 Perform element-by-element summation to obtain the attention map D′;

[0116] 6) Activate the attention graph D′ to obtain the activated attention graph E′; the process is: retain the values ​​of the corresponding positions of the two nodes with a connection relationship in the attention graph D′; set the values ​​of the corresponding positions of the two nodes without a connection relationship in the attention graph D′ to 0;

[0117] 7) Multiply element-by-element with the attention map E′ to obtain the feature matrix F′;

[0118] 8) Add the feature matrix F′ and the feature matrix A′ element by element to obtain the added feature matrix J′;

[0119] The feature matrix J′ is used as the output feature of the second graph self-attention mechanism GSA.

[0120] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0121] Specific implementation method seven: This implementation method is different from any one of specific implementation methods one to six in that: the pixel-based multi-attention fusion branch stage in step one includes a first channel attention (ChannelAttention) module, a first convolution block, a second convolution block, a first SE, a second SE, a second channel attention (ChannelAttention) module, a third convolution block, a fourth convolution block, and a third channel attention (ChannelAttention) module;

[0122] The first convolution block includes batch normalization BN, point-by-point convolution layer, ReLU activation function layer, BN, channel-by-channel convolution layer with convolution kernel size of 1, and ReLU activation function layer in sequence;

[0123] The second convolution block includes batch normalization BN, point-by-point convolution layer, ReLU activation function layer, BN, channel-by-channel convolution layer with a convolution kernel size of 5, and ReLU activation function layer in sequence;

[0124] The third convolution block includes batch normalization BN, point-by-point convolution layer, ReLU activation function layer, BN, channel-by-channel convolution layer with a convolution kernel size of 3, and ReLU activation function layer in sequence;

[0125] The fourth convolution block includes batch normalization BN, point-by-point convolution layer, ReLU activation function layer, BN, channel-by-channel convolution layer with a convolution kernel size of 7, and ReLU activation function layer in sequence.

[0126] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0127] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the specific working process of the pixel-based multi-attention fusion branch stage in step one is:

[0128] 1) Output features of the data preprocessing stage Input the first channel attention module, the first channel attention module outputs weighted data x′ (HSI pixel input X cnn Assign different spectral dimension weights so that the network can focus on key features in the data);

[0129] The specific process is:

[0130] The feature X output from the data preprocessing stage cnn Perform reshape deformation to obtain feature O;

[0131] Transpose feature O to obtain feature P;

[0132] Multiply feature O and feature P element by element to get feature Q, and then process feature Q with Softmax to get feature R.

[0133] Feature R is multiplied element by element by feature O to obtain feature U;

[0134] Reshape feature U to obtain feature V;

[0135] For feature V and feature X cnn Add element by element to get the feature first channel attention (ChannelAttention) module output weighted data x'; expressed as:

[0136]

[0137] Among them, d e Represents the features output from the data preprocessing stage The e-th element in d f Represents the features output from the data preprocessing stage The fth element in m e represents the weight of the first channel attention module,

[0138] x′ represents the weighted data after being processed by the first channel attention module, α represents the learnable coefficient preset to 0; Q represents the feature The total number of elements in

[0139] 2) Input the weighted data x′ into the first convolution block and the second convolution block respectively, and the first convolution block outputs the feature x 1 , the second convolutional block outputs feature x 2 ;

[0140] 3) The first convolutional block outputs feature x 1 Input the first SE, the first SE module outputs SE(x 1 ); The function of the SE module is to assign weights to spectra in different dimensions. Its detailed structure is as follows Figure 4 As shown. It is expressed as:

[0141] SE(x 1 )=Linear(Sigmoid(Linear(Avg(x 1 ))))×x 1

[0142] Among them, Avg(·) represents average pooling, Linear(·) represents linear mapping, and Sigmoid(·) represents activation function;

[0143] The second convolutional block outputs the feature x 2Input the second SE, the second SE module outputs SE (x 2 ); The function of the SE module is to assign weights to spectra in different dimensions. Its detailed structure is as follows Figure 4 As shown. It is expressed as:

[0144] SE(x 2 )=Linear(Sigmoid(Linear(Avg(x 2 ))))×x 2

[0145] 4) Output feature x of the first convolutional block 1 And the second convolutional block outputs the feature x 2 The fused features are obtained by element-by-element addition, and the fused features are input into the second channel attention module, which outputs weighted data.

[0146] 5) Add the weighted data output by the second channel attention module to the output of the first SE module SE(x 1 ) performs element-by-element addition to obtain the feature after element-by-element addition;

[0147] The element-wise added features are input into the third convolution block, and the third convolution block outputs the feature x 3 ;

[0148] 6) Add the weighted data output by the second channel attention module to the output of the second SE module SE(x 2 ) performs element-by-element addition to obtain the feature after element-by-element addition;

[0149] The element-wise added features are input into the fourth convolution block, and the fourth convolution block outputs the feature x 4 ;

[0150] 7) Output feature x of the third convolutional block 3 And the fourth convolutional block outputs feature x 4 Perform element-by-element addition to obtain the features after element-by-element addition;

[0151] The element-wise added features are input into the third channel attention module, and the third channel attention module outputs weighted data;

[0152] The third channel attention module outputs weighted data and the third convolutional block outputs feature x 3 And the fourth convolutional block outputs feature x4 Perform element-by-element addition to obtain the element-by-element added features, which are used as the output of the pixel-based multi-attention fusion branch stage.

[0153] GAT has some limitations in effectively utilizing local feature information. On the one hand, this is because GAT updates nodes through message passing between nodes, and this process considers more the relationships between nodes and the global context rather than the local details of the nodes. On the other hand, when processing large-scale graph data, the computational complexity and memory requirements of GAT increase dramatically, which further limits its ability to process local features.

[0154] In order to alleviate the above problems, a multi-attention fusion module (MAF) was built, and its detailed structure is as follows Figure 3 Specifically, first use Figure 4 ChannelAttention shown, for HSI pixel input X cnn Assigning different spectral dimension weights allows the network to focus on key features in the data.

[0155] After that, the weighted data is sent to two convolution blocks with similar structures but different functions. In the first layer of the convolution block, point-by-point convolution is used to integrate the information of different channels to ensure that the information between channels can be effectively circulated. The second layer uses channel-by-channel convolutions of different scales, which can not only learn spatial features under different receptive fields, but also extract features within the channel. Channel-by-channel convolution processes each channel independently, allowing the network to focus more on extracting unique and critical feature information from each channel. This design enables MAF to capture the local features of the data more accurately and effectively distinguish feature information of different importance. This process can be described as follows:

[0156] Among them, DW 1×1 (·) represents a channel-by-channel convolution block with a kernel size of 1; DW 5×5 (·) represents a channel-wise convolution block with a kernel size of 5; CA(·) represents channel attention.

[0157] Next, the output x of the previous layer 1 and x 2 They are sent to the SE modules in their respective branches. The function of the SE module is to assign weights to spectra in different dimensions. Its detailed structure is as follows: Figure 5 This process can be expressed as:

[0158] SE(x)=Linear(Sigmoid(Linear(Avg(x))))×x

[0159] Among them, Avg(·) represents average pooling, Linear(·) represents linear mapping, and Sigmoid(·) represents activation function. 1 and x 2 The features are fused and sent to the ChannelAttention module for secondary weighting. The weighted features will be used as shared weights and fused with the features weighted by the SE module on the two branches. This process enables the network to capture the local features of the image more accurately and distinguish feature information of different importance. This process can be expressed as:

[0160]

[0161] Then, we use point-wise convolution to integrate x again. 3 and x 4 The information of different channels in the network is combined with the larger channel-by-channel convolution blocks to extract local feature information from the network, thereby obtaining the output x 5 and x 6 These two convolution blocks can capture richer local details and further improve the representation ability of features. 3 and x 4 The fusion is sent to Channel Attention for the third weighting to enhance the weight of the key features. Finally, the weighted output is combined with x 5 and x 6 The fusion is then performed to obtain a more comprehensive and accurate feature representation. This process enables MAF to further capture and utilize local features in the image and improve the performance of the model. This process can be expressed as:

[0162] Among them, DW 3×3 (·) represents a channel-by-channel convolution block with a kernel size of 3; DW 7×7 (·) represents a channel-wise convolution block with a kernel size of 7.

[0163] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0164] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that: the classification stage in step 1 includes a Softmax classifier;

[0165] The specific working process of the classification stage is as follows:

[0166] The features output by the superpixel-based self-attention branch stage and the features output by the pixel-based multi-attention fusion branch stage are added element by element; expressed as:

[0167]

[0168] in, Represents the features output by the superpixel-based graph self-attention branch stage; represents the features output by the pixel-based multi-attention fusion branch stage; X out represents the result after fusion; β represents the weight coefficient of adaptive change;

[0169] The added features are input into the Softmax classifier, and the Softmax classifier outputs the classification result.

[0170] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0171] Specific implementation ten: This implementation differs from any one of specific implementations one to nine in that the training process of the greedy training strategy guided Graph self-attention network (GS-GraphSAT) in step one is:

[0172] 1) Segment the training set from the hyperspectral image (HSIs) data X Validation set and test set And from the training set X tr n% of the samples are split out again and temporarily merged with the validation set to obtain a new training set;

[0173] 2) Set the learning rate of the optimizer Adam to lr = 0.001, the training round epoch = 500, the superpixel segmentation scale scale = 500, the greedy training strategy GTS triggers the number of consecutive increases in loss n″ = 7, the current number of consecutive increases in loss n′ = 0, set the pre-convergence accuracy oa′, the number of sample return batches b = 10, and the current number of sample returns b′ = 0;

[0174] 3) Let the current number of iterations n = 1;

[0175] 4) Input the hyperspectral image training set into the graph self-attention network guided by the greedy strategy, and the graph self-attention network guided by the greedy strategy outputs the classification result and the corresponding graph self-attention network model guided by the greedy strategy;

[0176] 5) Input the validation set into the graph self-attention network model guided by the greedy strategy in 4), output the classification result based on the graph self-attention network model guided by the greedy strategy, calculate the accuracy of the classification result oa, save the accuracy of the current classification result oa and update the loss l of the graph self-attention network model guided by the greedy strategy n ′;

[0177] 6) If the accuracy of the current classification result oa≥oa′ and the current number of sample returns b′≤b, then mark the current model as having reached the "pre-convergence" state; otherwise, determine whether the number of iterations i has reached the training round epoch. If so, obtain the trained graph self-attention network model guided by the greedy strategy. If not, set the current number of iterations n=n+1 and continue to execute 4);

[0178] 7) If the current verification loss is l n ' -1 ≤l n ′, the current number of consecutive increases in loss n′ is incremented by 1; otherwise, determine whether the number of iterations i reaches the training round epoch. If so, obtain the trained graph self-attention network model guided by the greedy strategy. If not, set the current number of iterations n = n+1 and continue to execute 4);

[0179] 8) If the current loss increases continuously for n′=n″, take out the specified number of maximum loss samples from the validation set and merge them into the new training set; then, increment the number of sample returns b′ by 1 and reset n′ to zero; otherwise, determine whether the iteration number i reaches the training round epoch. If so, obtain the trained graph self-attention network model guided by the greedy strategy. If not, set the current iteration number n=n+1 and continue to execute 4).

[0180] This stage runs through the entire training process and enables the network to adaptively choose the best time to add samples to the training set, thereby more effectively utilizing sample resources and optimizing training results.

[0181] In the research of hyperspectral image classification methods, sample limitation is a common challenge. Due to the limited number of available samples, the training process of the classifier is often difficult to obtain sufficient data support, which directly leads to the inadequacy of the classifier performance, specifically the lack of classification accuracy and stability. Therefore, how to make more effective use of existing sample resources has become one of the key directions to improve the performance of hyperspectral image classification methods.

[0182] To address the above problems, a method called Greedy Training Strategy (GTS) is proposed. Its detailed workflow is shown in Algorithm 1. In the data preparation process, the data set is generally divided into training set X tr , validation set X va And the test set X te The training set X tr Used to train the model; validation set X va Used to evaluate the performance of the model and help determine whether the model is overfitting; test set X te Used to evaluate the final performance of the model. Unlike the previous method of setting the number of samples in the training set and the validation set to be equal, GTS further subdivides the training set. In the early stages of training, only a portion of the samples are used for network training, and the remaining number of samples are temporarily merged with the validation set. As training progresses, these samples will be reintegrated into the training set in batches at the appropriate time to achieve efficient use of sample resources and continuous improvement of model performance. The validation set plays an important role in evaluating model performance. Although it is not directly involved in training, it can help the model screen out samples with the greatest loss. These samples often contain feature information that the model has not yet understood. If the model can successfully "understand" and absorb this feature information, its ability to interpret the features will be significantly improved, thereby optimizing the overall performance of the model.

[0183] During the training of the model, when the network reaches an overall accuracy of 60% or more for the first time, it is considered to have reached the "pre-convergence" state, which indicates that the model has initially acquired the ability to understand features and process information. At this point, GTS begins to analyze the changes in loss during the model training process to accurately determine when to reintegrate the remaining number of samples into the training set. Specifically, when the model's loss no longer decreases within n consecutive epochs, it means that its classification performance has approached a maximum point, that is, the model has entered a convergence state. At this point, the verification process is used to screen out a predetermined number of maximum loss samples, and these samples are merged back into the training set. The addition of new samples may cause large fluctuations in training loss within the first few epochs. However, as the model converges again, its classification performance will be further improved.

[0184] During the training process guided by GTS, the loss change of the model will show such a periodic, fluctuating downward trend, which includes a cycle of decline and rise. Every time a new sample is merged into the training set, the model's ability to understand feature information will leap to a higher level. As all samples are gradually delivered, the model's classification performance will reach the optimal state. This process not only enriches the model's training data, but also enhances the model's ability to understand various features in the data set, thereby improving the overall performance of the model.

[0185] Regarding the appropriate time to deliver samples, it can be divided into two situations. The first is when the model first reaches the "pre-convergence" state, timely adding new samples to the training set can accelerate the convergence process of the model and help it quickly approach the optimal performance level. The second situation is when the model has reached the "optimal" state. At this time, if no new samples are added, the classification performance of the model tends to stabilize and it is difficult to improve significantly. However, it is at this stage that continuing to deliver new samples to the network enables it to continue to learn and capture more feature information, thereby further expanding its performance boundaries.

[0186] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0187] The following examples are used to verify the beneficial effects of the present invention:

[0188] Embodiment 1:

[0189] Next, we will verify the effectiveness of the proposed method GS-GraphSAT. This section will first introduce the datasets used in the experiment, the hardware configuration of the experiment, and the parameter settings. After that, we will conduct a comprehensive experimental verification of the proposed method from multiple aspects such as ablation experiments, quantitative evaluation, and visual evaluation.

[0190] A. Dataset Description

[0191] In order to verify the performance of the proposed method, this paper fully conducted experiments on three public datasets, namely Indian Pines, Pavia University, and WHU-Hi-LongKou.

[0192] The Indian Pines dataset is a hyperspectral image taken by AVIRIS in Indiana, USA in June 1992. It is 145×145 pixels in size and was used for early image classification research. Its wavelength range covers 400-2500 nanometers, with a spatial resolution of 20 meters and contains 220 bands. However, due to the presence of water absorption bands, the actual bands used for training are reduced to 200. The dataset covers 16 types of ground objects, including corn, oats, wheat, etc., with a total of 21,025 pixels, of which 10,249 are ground object pixels and the rest are background pixels.

[0193] The Pavia University dataset is derived from the hyperspectral images taken by the ROSIS-03 airborne imager in Pavia, Italy in 2003. It covers 115 bands in the wavelength range of 0.43-0.86μm with a spatial resolution of 1.3 meters. However, due to the influence of noise, 103 of the bands are usually used. The dataset contains 610×340 pixels, a total of 2247400 pixels, but only 42776 pixels represent ground objects, covering 9 types of ground objects, such as trees, asphalt roads, etc.

[0194] The WHU-Hi-LongKou dataset is a hyperspectral image data collected on July 17, 2018 in Longkou Town, Hubei Province, China, using a hyperspectral imaging sensor carried by a DJI Matrice 600Pro drone. The study area includes 9 crops, such as corn, cotton, narrow-leaf soybeans, and rice. The image size is 550×400 pixels, with a spatial resolution of about 0.463 meters, covering a wavelength of 400 to 1000 nanometers, and contains 270 bands.

[0195] Tables 1 to 3 list the main land cover categories involved in the three research scenarios mentioned above, as well as the number of training, validation, and test samples used for the classification tasks. Figures 6 to 8 The spatial distribution of training, validation, and testing samples for the three studied scenarios is shown.

[0196] Table 1 The name of each category in the Indian Pines dataset and the corresponding number of samples in the training set, validation set, and test set

[0197]

[0198] Table 2 The name of each category in the Pavia University dataset and the corresponding number of samples in the training set, validation set, and test set

[0199]

[0200] Table 3 Name of each category in the Longkou dataset and the corresponding number of samples in the training set, validation set, and test set

[0201]

[0202]

[0203] Experimental setup

[0204] Hardware configuration: The proposed GS-GraphSAT method is implemented in Pytorch1.10.1 and Python 3.7.0 on the Pycharm compiler software. In order to ensure the stability and efficiency of the experiment, this paper configures a high-performance hardware environment, including Intel Core i9-9900K CPU and NVIDIA GeForce RTX 3090GPU. At the same time, in order to avoid the accidental results, all experimental results are the average of 10 independent experiments.

[0205] Evaluation indicators: Three commonly used quantitative indicators are used to comprehensively evaluate the classification performance of each model, namely: overall accuracy (OA), average accuracy (AA) and Kappa coefficient (κ×100). Among them, OA measures the classification accuracy of the model on all samples, AA focuses on the average performance of the model on different categories, and the Kappa coefficient further measures the difference between the model classification results and random classification.

[0206] Parameter setting: In order to fairly evaluate the performance of the proposed method, all the methods used for comparison use the optimal hyperparameters recommended in the original paper. For the proposed method, we use the Adam optimizer to train the network with a learning rate of 1e-3, and the epoch is set to 500. The scales of the optimal superpixel segmentation on the IP, UP, and LK datasets are set to 100, 400, and 300, respectively, as shown in Table 4. In addition, Fig. 9 The figure shows the number of consecutive increases in loss required to trigger GTS. The optimal setting for the UP dataset is 5 times, while the optimal setting for the IP and LK datasets is 7 times.

[0207] Table 4 Comparison of different segmentation scales on three datasets

[0208]

[0209] Ablation experiment: The proposed method GS-GraphSAT contains three modules: GSA, MAF and GTS. In order to verify the effectiveness of each module, ablation experiments are carried out on three datasets, and OA is used as the main evaluation indicator. The experimental results are shown in Table 5.

[0210] Table 5 Ablation experiment

[0211]

[0212]

[0213] Specifically, this section designs a total of 8 groups of experiments to deeply explore the impact of each module on the overall performance. The first group of experiments uses the original GAT and the convolution branch composed of point-by-point convolution blocks and channel-by-channel convolution blocks, without introducing GTS. The results show that the classification performance on the three datasets is the lowest. The second to fourth groups of experiments respectively examine the situation when the three modules are used alone. The second group of experiments introduces GSA on the basis of the first group. Compared with the first group, the classification performance on the three datasets is improved by 0.61%, 0.45%, and 0.42%, respectively. This fully demonstrates the advantages of GSA in enhancing information transmission between nodes and mining node sequence attributes. The third group of experiments combines MAF with the original GAT. Compared with the first group, the classification performance on the three datasets is improved by 0.62%, 0.48%, and 0.56%, respectively. This proves the effectiveness of MAF in extracting local features at the pixel level. The fourth group of experiments introduces GTS on the basis of the first group. The classification performance on the three data sets was improved by 1.04%, 1.00%, and 0.57%, respectively. This improvement exceeded that of the first two groups, highlighting the key role of GTS in improving sample utilization efficiency. Groups 5 to 7 of experiments explored the effect of the combined use of the two modules. The experimental results show that the combined use of the two modules can further improve the classification performance, especially the combination of GTS and other modules, which shows significant performance gains. Finally, the eighth group of experiments integrated all three modules, and the classification performance of the network was optimal at this time. Compared with the first group, the performance improvements on the three data sets were 1.59%, 1.40%, and 0.97%, respectively. This not only proves that the three modules can work together, but also highlights the excellence and effectiveness of the proposed method GS-GraphSAT.

[0214] Quantitative experiments: In this section, three datasets are used to evaluate the performance of the proposed method and compared with eight advanced methods, including HybridSN, DTAN, SF, SSFTT, CEGCN, FDGC, WFCG, and AMGCFN. Specifically, HybridSN and DTAN are CNN-based methods; SF and SSFTT are Transformer-based methods; CEGCN, FDGC, WFCG, and AMGCFN are representatives of GNN and CNN hybrid methods. By analyzing the data in Tables 6 to 8, it can be found that on the three datasets of IP, UP, and LK, the proposed method shows significant advantages in key indicators such as OA, AA, and Kappa.

[0215] Specifically, in the CNN-based method, although HybridSN combines 2D and 3D convolutions, its classification performance is slightly insufficient due to the lack of the introduction of the attention mechanism. DTAN introduces an attention module to enable the network to more accurately capture the importance of different features, so its performance is better than HybridSN. In the Transformer-based method, SF mainly focuses on spectral dimension features and pays insufficient attention to the spatial dimension, resulting in poor performance. SSFTT, on the other hand, combines convolution and feature markers to extract primary and high-level semantic features, achieving relatively good classification results.

[0216] The hybrid method of GNN and CNN has shown certain advantages in this field. CEGCN was the first to combine GCN and CNN to achieve the fusion of superpixels and pixels, and achieved relatively good performance. The subsequent WFCG and AMGCFN as well as the method proposed in this article all adopted this structure. WFCG further improved its performance by introducing GAT and attention modules. AMGCFN, by introducing multi-hop graph mechanism and multi-scale CNN, provided new ideas for research in this field and achieved excellent performance. In contrast, FDGC's classification performance is slightly inferior because it does not adopt the method of superpixel and pixel fusion and has a relatively simple structure.

[0217] Next, we analyze the method proposed in this paper. On the IP dataset, compared with the best comparison method, the proposed method GS-GraphSAT is 1.26%, 1.66% and 1.44% higher in three key indicators. In the optimal category, GS-GraphSAT performed even better, with 15 of the 16 categories achieving the best. On the UP dataset, GS-GraphSAT also showed its advantages, with the three indicators improved by 0.84%, 1.50% and 1.12% respectively compared with the best comparison method. In the optimal category, GS-GraphSAT also basically maintained a high accuracy of more than 99%. On the LK dataset, GS-GraphSAT is also not inferior, with the three indicators higher than the best comparison method by 0.59%, 1.53% and 0.78% respectively. In the optimal category, 6 out of 9 categories are in the leading position.

[0218] Table 6 Classification performance of different methods on the IndianPines dataset (the best results are in bold)

[0219]

[0220] Table 7 Classification performance of different methods on Pavia University dataset (best results are in bold)

[0221]

[0222] Table 8 Classification performance of different methods on Longkou dataset (the best results are in bold)

[0223]

[0224]

[0225] In order to more comprehensively evaluate the performance of the proposed method, this paper also counted the average time consumption of a single training and test of each method. The results are shown in Table 9. The method using superpixels and pixel fusion consumes significantly less time than other methods, which is mainly due to its full pixel input method, which does not require batch training. Among the methods based on the hybrid of GNN and CNN, although FDGC consumes the most time, its performance is the lowest. CEGCN has the lowest training time consumption because it does not introduce an attention module. AMGCFN performs best in test time and its performance is relatively excellent. Although the proposed method is slightly inferior in training time, the performance advantage it brings is obvious. The comparison of these experimental data fully demonstrates the superiority of the proposed method in classification performance.

[0226] Table 9 Comparison of single training and testing time of different methods on three datasets

[0227]

[0228] Visual Assessment

[0229] a) Classification chart comparison: In this part, Figure 10 to Figure 12 The classification graphs of all methods on three datasets are shown. It can be observed that the classification graphs of CNN-based and Transformer-based methods are relatively inferior to those of GNN-CNN hybrid methods. This may be because the former two have limitations in processing sample feature relationships and fail to fully capture and utilize the intrinsic connections between samples, thus restricting their classification performance to a certain extent.

[0230] Although the hybrid methods based on GNN and CNN have attempted to solve the above problems, their classification graphs still show a lot of misclassification. However, compared with these methods, the proposed method shows significant advantages. Its classification graph retains complete information while significantly reducing the number of misclassifications, which fully demonstrates the richness and accuracy of this method in feature representation learning.

[0231] b) Confusion matrix comparison: In this part, this paper compares the confusion matrix between the best method among the three comparison methods and the proposed method. The results are as follows Fig.13 As shown in the figure. The confusion matrices of the proposed method on the three datasets show the least misclassification. In contrast, the CNN-based DTAN and the Transformer-based SSFTT both show more misclassifications, which to some extent reveals the deficiencies of these two networks in learning feature information. At the same time, the WFCG method based on a mixture of GNN and CNN performs better than the former two, which once again proves the effectiveness of using superpixels and pixel structures. However, due to the limitations of the GAT branch of WFCG in information transmission between nodes, and the insufficiency of its CNN branch in extracting local features at the pixel level, its confusion matrix results are still inferior to those of the proposed method.

[0232] c) Heatmap Comparison: In this section, this paper visualizes the heatmaps of each module of the proposed method to further demonstrate its effectiveness. Fig.14 As shown, these heatmaps provide an intuitive visual comparison. Fig.14Part (a) in the figure shows the results of the initial serial CNN. By comparison, it can be found that the initial CNN does not fully capture local details when extracting features, and the response in the heat map is relatively weak. Then observe part (b), which is the result of the proposed MAF module. It can be clearly seen that MAF performs well in extracting local detail features, and the thermal response is significantly enhanced. This result proves the effectiveness of MAF in extracting local detail features at the pixel level. Let's look at part (c), which is the result of GSA based on superpixel nodes. GSA pays more attention to the smoothness of global features, but is slightly insufficient in capturing local features. This is exactly the original intention of MAF, which aims to make up for this shortcoming. Finally, observe part (d), which is the result after the fusion of GSA and MAF. The heat map at this time not only shows clear local features, but also integrates global context information, so that the network can learn the feature representation in the input data more comprehensively and accurately. This result fully demonstrates the advantages of the complementary fusion of GSA and MAF. These heat map visualization results further prove the role of each module proposed in the feature extraction process and once again verify the effectiveness of the proposed method.

[0233] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A hyperspectral image classification method based on a graph self-attention network guided by a greedy strategy, characterized by: The specific process of the method is: Step 1: Construct a graph self-attention network guided by a greedy strategy; The graph self-attention network guided by the greedy strategy includes a data preprocessing stage, a superpixel-based graph self-attention branch stage, a pixel-based multi-attention fusion branch stage, a classification stage, and a greedy training strategy stage. The data preprocessing stage consists of two parts: 1) Preliminary preparation of hyperspectral image data: Linear discriminant analysis LDA is used to extract the mid-band of hyperspectral image X; The extracted bands are segmented into superpixels using the simple linear iterative clustering (SLIC) method to obtain the relationship matrix And the adjacency matrix H and W represent the height and width of HSI respectively; N represents the number of graph nodes; represents real numbers; The relationship matrix G is expressed as follows: Among them, G i,j Represents the value of the relationship matrix at (i, j), i represents the i-th pixel, j represents the j-th superpixel; the relationship matrix G i,j The rows represent pixels and the columns represent superpixels; Flatten(·) represents the flattening operation. express The i-th pixel in S j represents the jth superpixel in S; represents a one-dimensional vector after flattening the spatial dimension; S represents a superpixel; The adjacency matrix H is represented by the following formula: Among them, H i′,j′ represents the value of the adjacency matrix at (i′, j′), i′ represents the i′th superpixel, and j′ represents the j′th superpixel; the rows and columns of the adjacency matrix H represent superpixels, and the elements in the adjacency matrix H represent whether there is a connection between nodes (i′, j′); S i′ represents the i′th superpixel in S, S j′ represents the j′th superpixel in S; 2) The hyperspectral image X is subjected to two point-by-point convolution operations to obtain the output features It is expressed as: X cnn =PW(PW(X)) Where PW(·) represents a point-wise convolution block, which includes a batch normalization layer, a convolution layer, and a ReLU activation function layer in sequence; L represents the length of the spectral sequence after two point-wise convolution operations: The specific working process of the graph self-attention network guided by the greedy strategy is as follows: The hyperspectral image is input into the data preprocessing stage. The output image of the data preprocessing stage is input into the superpixel-based image self-attention branch stage and the pixel-based multi-attention fusion branch stage respectively. The features output by the superpixel-based image self-attention branch stage and the pixel-based multi-attention fusion branch stage are fused. The fused features are input into the classification stage, and the classification stage outputs the classification results. Step 2: Classify the hyperspectral image to be tested based on the graph self-attention network guided by the greedy strategy to obtain the classification result.

2. The hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy according to claim 1 is characterized by: The superpixel-based graph self-attention branch stage in step one includes a coding layer, a first graph self-attention mechanism GSA, a second graph self-attention mechanism GSA, and a decoding layer in sequence.

3. The hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy according to claim 2 is characterized by: The working process of the superpixel-based graph self-attention branch stage in step 1 is as follows: 1) Output features of the data preprocessing stage and the relationship matrix G are input into the encoding layer, and the encoding layer outputs the transformed graph The process is expressed as: Among them, V represents the set of graph nodes composed of superpixels; represents the relationship matrix after column regularization of the relationship matrix G; the superscript T represents the transposition; Flatten(X cnn ) means flattening X; Encode(·) means encoding; 2) Figure The adjacency matrix H output in the data preprocessing stage is input into the first graph self-attention mechanism GSA, and the first graph self-attention mechanism GSA outputs features; the first graph self-attention mechanism GSA output features and the adjacency matrix H output in the data preprocessing stage are input into the second graph self-attention mechanism GSA, and the second graph self-attention mechanism GSA outputs a weighted graph; 3) Input the weighted graph and relationship matrix G into the decoding layer, and the decoding layer outputs pixel data X1; the process is expressed as: X1=Decode(V′;G)=Reshape(G×V′) Among them, Decode(·) is used to decode node data into pixel data; Reshape(·) is used to restore the spatial dimension of flattened data; V′ represents the weighted graph node set output by the second graph self-attention mechanism GSA.

4. The hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy according to claim 3 is characterized by: 2) Nodes in the graph The adjacency matrix H output in the data preprocessing stage is input into the first graph self-attention mechanism GSA, and the first graph self-attention mechanism GSA outputs features; the specific working process is: (1) Use mapping operations to input graph data Transform into graph data as well as Where L = h × d, h represents the number of multiple heads, and d represents the length of the sequence after multiple head segmentation; right and Perform matrix multiplication to obtain the attention map M, and use the Softmax function to normalize the attention map M and combine it with Perform matrix multiplication to get the output result The expression is: The superscript T indicates transposition; d k express Dimensions; (2) Output results of multiple attention heads The final result A is obtained by cascading; (3) Accumulate the attention maps M of multiple heads to obtain the output M1; (4) Graph nodes Input the graph attention network GAT, and the graph attention network GAT outputs the attention map M2; (5) Add the attention map M1 and the attention map M2 element by element to obtain the attention map D; (6) Activate the feature matrix D to obtain the activated feature matrix E. The process is as follows: retain the values ​​of the corresponding positions of the two nodes in the feature matrix D that have a connection relationship; set the values ​​of the corresponding positions of the two nodes that have no connection relationship in the feature matrix D to 0; (7) Multiply element-by-element with the feature matrix E to obtain the weighted graph F; (8) Add the weighted graph A and the weighted graph F element by element to obtain the summed graph J; (9) Repeat (1) to (8) K times, concatenate the feature matrix results obtained K times, and use the concatenation result as the output feature of the first graph self-attention mechanism GSA.

5. The hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy according to claim 4 is characterized by: The value of K is 4.

6. The hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy according to claim 5 is characterized by: In the above 2), the output features of the first graph self-attention mechanism GSA and the adjacency matrix H output in the data preprocessing stage are input into the second graph self-attention mechanism GSA, and the second graph self-attention mechanism GSA outputs a weighted graph; the specific process is: 1) Use the mapping operation to output the graph data of the first graph self-attention mechanism GSA Transform into graph data as well as right and Perform matrix multiplication to obtain the attention map M′, and use the Softmax function to normalize the attention map M′ and combine it with Perform matrix multiplication to get the output result The expression is: The superscript T means to find the transpose; 2) Output results of multiple heads Cascade to obtain the feature matrix A′; 3) Accumulate the attention maps M′ of multiple heads to obtain the attention map M′1; 4) Figure Input the graph attention network GAT, and the graph attention network GAT outputs the attention map M′2; 5) Add the attention map M′1 and the attention map M′2 element by element to obtain the attention map D′; 6) Activate the attention graph D′ to obtain the activated attention graph E′; the process is: retain the values ​​of the corresponding positions of the two nodes with a connection relationship in the attention graph D′; set the values ​​of the corresponding positions of the two nodes without a connection relationship in the attention graph D′ to 0; 7) Multiply element-by-element with the attention map E′ to obtain the feature matrix F′; 8) Add the feature matrix F′ and the feature matrix A′ element by element to obtain the added feature matrix J′; The feature matrix J′ is used as the output feature of the second graph self-attention mechanism GSA.

7. The hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy according to claim 6 is characterized by: The pixel-based multi-attention fusion branch stage in the step 1 includes a first channel attention module, a first convolution block, a second convolution block, a first SE, a second SE, a second channel attention module, a third convolution block, a fourth convolution block, and a third channel attention module; The first convolution block includes batch normalization BN, point-by-point convolution layer, ReLU activation function layer, BN, channel-by-channel convolution layer with convolution kernel size of 1, and ReLU activation function layer in sequence; The second convolution block includes batch normalization BN, point-by-point convolution layer, ReLU activation function layer, BN, channel-by-channel convolution layer with a convolution kernel size of 5, and ReLU activation function layer in sequence; The third convolution block includes batch normalization BN, point-by-point convolution layer, ReLU activation function layer, BN, channel-by-channel convolution layer with a convolution kernel size of 3, and ReLU activation function layer in sequence; The fourth convolution block includes batch normalization BN, point-by-point convolution layer, ReLU activation function layer, BN, channel-by-channel convolution layer with a convolution kernel size of 7, and ReLU activation function layer in sequence.

8. The hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy according to claim 7 is characterized by: The specific working process of the pixel-based multi-attention fusion branch stage in step 1 is as follows: 1) Output features of the data preprocessing stage Input the first channel attention module, and the first channel attention module outputs weighted data x′; The specific process is: The feature X output from the data preprocessing stage cnn Perform reshape deformation to obtain feature O; Transpose feature O to obtain feature P; Multiply feature O and feature P element by element to get feature Q, and then process feature Q with Softmax to get feature R. Feature R is multiplied element by element by feature O to obtain feature U; Reshape feature U to obtain feature V; For feature V and feature X cnn Perform element-by-element addition to obtain the feature first channel attention module output weighted data x′; 2) Input the weighted data x′ into the first convolution block and the second convolution block respectively. The first convolution block outputs feature x1, and the second convolution block outputs feature x2; 3) The first convolutional block outputs feature x1 which is input into the first SE. The first SE module outputs SE(x1); it is expressed as: SE(x1)=Linear(Sigmoid(Linear(Avg(x1))))×x1 Among them, Avg(·) represents average pooling, Linear(·) represents linear mapping, and Sigmoid(·) represents activation function; The second convolutional block outputs feature x2 which is input into the second SE. The second SE module outputs SE(x2); it can be expressed as: SE(x2)=Linear(Sigmoid(Linear(Avg(x2))))×x2 4) Add the output feature x1 of the first convolution block and the output feature x2 of the second convolution block element by element to obtain the fused feature, input the fused feature into the second channel attention module, and the second channel attention module outputs weighted data; 5) Add the weighted data output by the second channel attention module and the output SE(x1) of the first SE module element by element to obtain the feature after element-by-element addition; The element-wise added features are input into the third convolution block, and the third convolution block outputs feature x3; 6) Add the weighted data output by the second channel attention module and the output SE(x2) of the second SE module element by element to obtain the feature after element-by-element addition; The element-wise added features are input into the fourth convolution block, and the fourth convolution block outputs feature x4; 7) Add the output feature x3 of the third convolution block and the output feature x4 of the fourth convolution block element by element to obtain the feature after element-by-element addition; The element-wise added features are input into the third channel attention module, and the third channel attention module outputs weighted data; The weighted data output by the third channel attention module, the output feature x3 of the third convolution block, and the output feature x4 of the fourth convolution block are added element by element to obtain the element-by-element added features, which are used as the output of the pixel-based multi-attention fusion branch stage.

9. The hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy according to claim 8 is characterized by: The classification stage in step 1 includes a Softmax classifier; The specific working process of the classification stage is as follows: The features output by the superpixel-based self-attention branch stage and the features output by the pixel-based multi-attention fusion branch stage are added element by element; expressed as: in, Represents the features output by the superpixel-based graph self-attention branch stage; represents the features output by the pixel-based multi-attention fusion branch stage; X out represents the result after fusion; β represents the weight coefficient of adaptive change; The added features are input into the Softmax classifier, and the Softmax classifier outputs the classification result.

10. The hyperspectral image classification method based on the graph self-attention network guided by the greedy strategy according to claim 9 is characterized by: The training process of the graph self-attention network guided by the greedy strategy in step 1 is: 1) Segment the training set from the hyperspectral image data X Validation set and test set And from the training set X tr n% of the samples are split out again and temporarily merged with the validation set to obtain a new training set; 2) Set the learning rate of the optimizer Adam to lr = 0.001, the training round epoch = 500, the superpixel segmentation scale scale = 500, the greedy training strategy GTS triggers the number of consecutive increases in loss n″ = 7, the current number of consecutive increases in loss n′ = 0, set the pre-convergence accuracy oa′, the number of sample return batches b = 10, and the current number of sample returns b′ = 0; 3) Let the current number of iterations n = 1; 4) Input the hyperspectral image training set into the graph self-attention network guided by the greedy strategy, and the graph self-attention network guided by the greedy strategy outputs the classification result and the corresponding graph self-attention network model guided by the greedy strategy; 5) Input the validation set into the graph self-attention network model guided by the greedy strategy in 4), output the classification result based on the graph self-attention network model guided by the greedy strategy, calculate the accuracy of the classification result oa, save the accuracy of the current classification result oa and update the loss l of the graph self-attention network model guided by the greedy strategy n ′; 6) If the accuracy of the current classification result oa≥oa′ and the current number of sample returns b′≤b, then the current model is marked as having reached the "pre-convergence" state; Otherwise, determine whether the number of iterations i reaches the training round epoch. If so, obtain the trained graph self-attention network model guided by the greedy strategy. If not, set the current number of iterations n = n + 1 and continue to execute 4); 7) If the current verification loss is l n ' -1 ≤l n ′, then the number of consecutive increases in the current loss n′ will be incremented by 1; Otherwise, determine whether the number of iterations i reaches the training round epoch. If so, obtain the trained graph self-attention network model guided by the greedy strategy. If not, set the current number of iterations n = n + 1 and continue to execute 4); 8) If the current loss increases continuously for n′=n″, take out the specified number of maximum loss samples from the validation set and merge them into the new training set; then, add 1 to the number of sample returns b′ and reset n′ to zero; Otherwise, determine whether the number of iterations i reaches the training round epoch. If so, obtain the trained graph self-attention network model guided by the greedy strategy. If not, set the current number of iterations n=n+1 and continue to execute 4).

Images