Parallel dynamic multi-hop graph and composite multi-scale convolution hyperspectral sparse unmixing network

Abstract

The invention relates to the technical field of hyperspectral image processing, and discloses a parallel dynamic multi-hop graph and composite multi-scale convolution hyperspectral sparse unmixing network, which comprises a dynamic multi-hop graph interactive attention module (DMGIAM) and a composite multi-scale convolution space-spectral attention module (CMS2AM). The DMGIAM captures long-distance spatial dependency features through dynamic jump perception graph convolution and a cross-head interactive attention mechanism; the CMS2AM extracts local spectrum-space features through a multi-scale convolution residual error and a double-convolution attention module. In addition, the invention further provides a self-adaptive weighted total variation loss function (SGATV) based on a Sobel operator and a Gaussian function, noise is effectively restrained, and edge details are reserved. According to the parallel dynamic multi-hop graph and the composite multi-scale convolution hyperspectral sparse unmixing network provided by the invention, the defects of an existing method in the aspects of long-distance feature capture, multi-scale information fusion and calculation efficiency are overcome, and the precision and robustness of hyperspectral unmixing are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image processing, and particularly to a parallel dynamic multi-hop graph and a composite multi-scale convolutional hyperspectral sparse unmixing network. Background Art

[0002] With the rapid development of hyperspectral imaging technology, hyperspectral imaging sensors can capture more detailed and abundant spectral information, which provides broad prospects for the application of hyperspectral images (HSIs) in multiple fields. However, limited by the spatial resolution of the sensor, a single pixel usually contains the mixed spectra of multiple ground objects (i.e., "mixed pixels"), seriously affecting the classification and detection accuracy. Hyperspectral unmixing (HU) aims to decompose mixed pixels into pure endmembers and their abundances, which is a key technology to improve data availability.

[0003] Existing unmixing methods are mainly based on the linear spectral mixture model (LMM) and the nonlinear model (NLMM). LMM is widely used because of its simplicity and efficiency, and has derived geometric methods (such as VCA, MVSA), statistical methods (such as NMF), sparse regression methods, and tensor decomposition methods. Geometric methods rely on convex geometry theory, but are limited by the pure pixel assumption; NMF-based methods can handle mixed data, but their non-convexity makes them prone to falling into local optima, and regularization terms (such as L1 / 2 sparsity, total variation TV, etc.) need to be designed manually, and the process is cumbersome. Although tensor decomposition can retain the spatial spectral structure, the determination of the tensor rank lacks an effective standard, which limits its practicality.

[0004] Sparse regression methods search for endmember subsets through spectral libraries, avoiding endmember extraction. However, TV regularization only acts on the spatial domain, resulting in the decoupling of space and spectrum and high computational complexity. In addition, traditional methods rely on numerical optimization, and the unmixing results are sensitive to initialization and lack efficient non-linear feature learning capabilities.

[0005] In recent years, neural networks have shown outstanding performance in hyperspectral unmixing due to their advantages in non-linear feature extraction. Autoencoders (AEs) estimate abundances through low-dimensional representations, but pixel-by-pixel training ignores spatial information. AE methods combined with convolutional neural networks (CNNs) (such as MiSiCNet, RDAE) use local features to improve performance, but CNNs can only capture short-range spatial dependencies. Graph convolutional networks (GCNs) combined with superpixel segmentation (such as GACAE) expand the receptive field, but the number of superpixels limits the information extraction distance, and the computational complexity increases sharply as the network deepens. Some methods (such as SUnCNN) introduce spectral libraries to enhance unmixing, but still face the problem of insufficient modeling of long-range pixel relationships.

[0006] Although the methods in the prior art perform excellently in capturing local features, there are still deficiencies in the feature extraction of long-range spatial dependencies. Some methods expand the capture range of spatial information through techniques such as superpixel segmentation, but the improvement in the information extraction distance is still limited. Although the receptive field can be expanded by increasing the convolution kernels of CNN and GCN or deepening the network to capture long-distance pixel dependencies, as the network depth increases, the model performance often degrades, accompanied by a significant increase in computational complexity and waste of resources. This has led most of the existing methods into a dilemma: to expand the receptive field and capture more long-distance dependency relationships, the network needs to become deeper; but at the same time, to maintain good performance and computational efficiency, the network cannot be overly complex. Therefore, how to ensure that the network effectively captures long-distance features while controlling and reducing computational costs and resource consumption has become a key problem in current research.

[0007] Therefore, it is necessary to provide a parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network to solve the above technical problems. Summary of the Invention

[0008] The present invention provides a parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network, aiming to ensure that the network effectively captures long-distance features while controlling and reducing computational costs and resource consumption.

[0009] The parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network provided by the present invention includes a dynamic multi-hop graph interaction attention module (DMGIAM) and a composite multi-scale convolutional spatial-spectral attention module (CMS2AM);

[0010] Among them, the dynamic multi-hop graph interaction attention module (DMGIAM) is used to extract long-distance spatial dependency features of hyperspectral images, including two sub-modules: dynamic multi-hop perception graph convolution (DMPGC) and cross-head interaction attention (CHIA);

[0011] The dynamic multi-hop perception graph convolution (DMPGC) generates an adjacency matrix through superpixel segmentation and adaptively weights and aggregates graph features of different hop numbers;

[0012] The cross-head interaction attention (CHIA) fuses multi-hop features through a multi-head attention mechanism and a cross-head interaction layer;

[0013] Among them, the composite multi-scale convolutional spatial-spectral attention module (CMS2AM) is used to extract local multi-scale spectral-spatial features, including a composite multi-scale convolutional residual module (CMSCR) and a double convolutional spatial-spectral attention module (DCS2A);

[0014] The Composite Multi-Scale Convolutional Residual Module (CMSCR) extracts local features of different scales through multi-scale convolutional kernels and performs residual fusion;

[0015] The Dual-Convolution Spatial-Spectral Attention Module (DCS2A) enhances feature representation through spectral attention and spatial attention mechanisms;

[0016] The network structure includes an Adaptive Weighted Total Variation Loss Function (SGATV) for dynamically adjusting weights by combining the Sobel edge detection operator and Gaussian function to promote the smoothing and denoising of the abundance matrix; the parallel structure of DMGIAM and CMS2AM inputs the dimensionality-reduced hyperspectral data and the original data respectively, and outputs the abundance matrix through feature splicing and fusion.

[0017] Preferably, the Dynamic Multi-hop Perception Graph Convolution (DMPGC) specifically includes:

[0018] Generating a pixel-superpixel transformation matrix (B), an adjacency matrix (A), and a superpixel information matrix (S) using SLIC superpixel segmentation;

[0019] Generating a multi-hop adjacency matrix through the Depth-First Search (DFS) algorithm and calculating weights using a Gaussian kernel function;

[0020] Adopting a Weighted Aggregation (WA) module to dynamically adjust the weights of features with different hop numbers and generate fused multi-hop graph features.

[0021] Preferably, the Cross-Head Interaction Attention (CHIA) specifically includes:

[0022] Dividing the input features into multiple attention heads and generating query (Q), key (K), and value (V) matrices respectively;

[0023] Implementing feature interaction and information sharing between different attention heads through a cross-attention mechanism;

[0024] Using layer normalization (LayerNorm) and residual connections to improve training stability.

[0025] Preferably, the Composite Multi-Scale Convolutional Residual (CMSCR) specifically includes:

[0026] Using a 1×1 convolutional kernel to reduce the parameter dimension and parallelly extracting multi-scale features with 3×3, 5×5, and 7×5 convolutional kernels;

[0027] Adding the multi-scale features and splicing them with the original input, and adjusting the output channel number through a 1×1 convolution.

[0028] Preferably, the Dual-Convolution Spatial-Spectral Attention (DCS2A) specifically includes:

[0029] Spectral Attention Module: Generates channel weights through global average pooling and max pooling to enhance the features of key spectral channels;

[0030] Spatial Attention Module: Generates spatial weights through channel - dimension pooling and cascaded convolution to enhance the features of key spatial positions;

[0031] Fuses the spectral and spatial attention weights with the input features to generate an enhanced feature map.

[0032] Preferably, the construction of the Adaptive Weighted Total Variation Loss Function (SGATV) includes:

[0033] Calculates pixel gradients using the Sobel operator and generates an adaptive weight matrix in combination with the Gaussian function;

[0034] Combines the weight matrix with the Total Variation (TV) loss to retain edge details while smoothing noise.

[0035] Preferably, the network adopts a parallel structure, splices the long - distance features output by DMGIAM and the multi - scale local features output by CMS2AM, and outputs the abundance matrix after fusion through two - layer CBLM modules.

[0036] Preferably, the network is trained by the Adam optimizer, the learning rate is set to le - 3, and the total loss function is the weighted sum of the Mean Square Error (MSE) and the SGATV loss.

[0037] Compared with the related technologies, the parallel dynamic multi - hop graph and composite multi - scale convolutional hyperspectral sparse unmixing network provided by the present invention have the following beneficial effects:

[0038] The present invention uses a dynamic multi - hop graph interaction attention module to capture long - distance spatial - spectral information, and utilizes multi - scale convolution and spectral - spatial attention to fuse features, realizing the fine processing of rich spatial - spectral information at multiple scales in the local area. Moreover, the parallel architecture design proposed by the present invention improves the computational efficiency of the model;

[0039] The present invention protects its architectural innovation as a neural network model and the electrical connection and data transfer relationship between its modules by introducing a parallel structure of the dynamic multi - hop graph interaction attention module (DMGIAN) and the composite multi - scale convolutional spatial - spectral attention module (MACSAF); this model realizes the efficient processing of capturing long - distance features and aggregating multi - scale features in the hyperspectral image sparse unmixing task; the present invention is applicable to hyperspectral image unmixing, remote sensing data analysis, and other application scenarios that require sparse feature extraction. Brief Description of the Drawings

[0040] Figure 1Schematic diagram of a parallel dynamic multi-hop graph and a composite multi-scale convolutional hyperspectral sparse unmixing network provided by the present invention Figure 1 The symbols in it are as follows:

[0041] A: Adjacency matrix, S: Superpixel information matrix, B: Pixel-to-superpixel transformation matrix, BN: Batch normalization operation, DFS: Depth-first traversal operation, i-hop: i-hop adjacency matrix, WA: Weight matrix, GCNB: Graph convolution sub-module, Concat: Matrix concatenation operation, CHIA: Cross-head interaction attention module, CBLM: Composite sub-module, CNNB: Convolution sub-module, DCS2A: Dual convolutional spatial-spectral attention module;

[0042] Figure 2 Structural diagrams of (a) CBLM, (b) CNNB, and (c) GCNB Figure 2 The symbols in it are as follows:

[0043] K: Convolution kernel size, DP: Dropout network layer, BN: Batch normalization layer, WA: Weight self-learning matrix, CNN: Convolution, GCN: Graph convolution, LReLu: Leaky ReLU network layer, AvgPool: Average pooling;

[0044] Figure 3 Schematic diagram of the CHIA model. The symbols in the figure are as follows:

[0045] Matrix multiplication Matrix addition, DP: Dropout network layer, LReLu: Leaky ReLU network layer, HIL: Head interaction layer;

[0046] Figure 4 Schematic diagram of the DCS2A model. The symbols in the figure are as follows:

[0047] Matrix multiplication Matrix addition, LReLu: Leaky ReLU network layer, AvgPool: Average pooling layer, MaxPool: Max pooling layer, BN: Batch normalization layer;

[0048] Figure 5 Endmember curves and images of DC1 and DC2: (a) Endmember curve of DC1, (b) Endmember curve of DC2, (c) Pseudo-color image of DC1, (d) Pseudo-color image of DC2;

[0049] Figure 6 True-color images of four real datasets, (a) JasperRidge, (b) Samson, (c) Cuprite, (d) Urban

[0050] Figure 7 The abundance images of the DC1 dataset at three signal-to-noise ratios, where GT is the true abundance image;

[0051] Figure 8 The abundance images of the DC2 dataset at three signal-to-noise ratios, where GT is the true abundance image;

[0052] Figure 9 The abundance images generated by the model of the present invention for the Jasper dataset;

[0053] Figure 10 The abundance images generated by the model of the present invention for the Samson dataset;

[0054] Figure 11 The abundance images generated by the model of the present invention for the Cuprite dataset;

[0055] Figure 12 The abundance images generated by the model of the present invention for the Urban dataset. Detailed implementation manners

[0056] The neural network model of the present invention includes a dynamic multi-hop graph interaction attention module (DMGIAN) and a composite multi-scale convolutional spatial-spectral attention module (MACSAF). The two modules are in a parallel structure, processing the input hyperspectral data separately and fusing at the end. Among them, the input of DMGIAN is the dimension-reduced HSI data, while the input of MACSAF is the original HSI data. The output of the network is an abundance matrix, which is multiplied by the spectral library matrix and then fitted with the original HSI. In this process, the loss function plays a role in regularization, and then the model parameters are updated using the forward feedback mechanism to achieve the accurate extraction of the abundance image. The electrical connection relationship between the modules is reflected in data flow and tensor transfer.

[0057] The present invention will be described in detail below in conjunction with the accompanying drawings and implementation manners.

[0058] The network architecture of the present invention has been given in Figure 1 . Generally speaking, the network consists of two main components: A. The dynamic multi-hop graph interaction attention module (DMGIAM) and B. The composite multi-scale convolutional spatial-spectral attention module (CMS2AM). First, the PCA-dimension-reduced HSI and the original HSI are respectively input into DMGIAM and CMS2AM. Subsequently, the present invention integrates the convolutional module, batch normalization (BN) block, and LeakyReLU (LReLU) activation into a unified module, called CBLM (as shown in Figure 2(as shown in (a)) to enhance feature extraction and improve model performance. In addition, the long-range dependence features captured by DMGIAM are concatenated with the multi-scale local spectral features generated by CMS2AM and the 4-channel features output by the CBLM block. Finally, these features are further fused through two layers of CBLM to obtain the output feature matrix corresponding to the abundance matrix. The network also proposes an adaptive weighted TV loss function (SGATV) based on the Sobel operator and Gaussian function. This improved TV loss function promotes the smoothing and denoising of the abundance matrix in the forward feedback mechanism, thus significantly improving the unmixing performance of HSI. Next, the present invention will introduce these two main components in detail:

[0059] A. Dynamic Multi-hop Graph Interaction Attention Module (DMGIAM)

[0060] The original multi-hop graph network can capture the dependencies between distant nodes through the multi-hop mechanism, enabling the model to consider a wider range of context information. However, it does not consider the weight differences between different hop numbers when aggregating multi-hop features, resulting in the underutilization of key information and thus affecting the overall performance of the model.

[0061] The multi-head attention mechanism can independently learn different subspace features through multiple parallel attention heads. However, each of its heads operates independently. Although it can learn different features, there is a lack of collaboration between the heads, resulting in the model being unable to effectively distinguish the spatial dependencies between distant and close ranges when processing multi-hop features, causing the features of different hop numbers to overlap in the feature space and unable to exert the advantages of the multi-hop mechanism, thereby reducing the overall expressive ability of the model and its sensitivity to detailed information.

[0062] Based on the above problems, the present invention designs DMGIAN to extract the long-distance spatial features of hyperspectral images. This network consists of two sub-modules: Dynamic Multi-hop Perception Graph Convolution (DMPGC) and Cross-head Interaction Attention (CHIA).

[0063] 1) Dynamic Multi-hop Perception Graph Convolution (DMPGC)

[0064] DMPGC is optimized based on the multi-hop graph structure. Through the designed WA module, this model can weight-aggregate the features extracted by each GCN in an autonomous learning manner. Moreover, the WA module can also adaptively integrate and fuse the information between multi-hop feature images. Compared with the original multi-hop graph network, DMPGC can more accurately distinguish important features from secondary features by dynamically adjusting the weights between different levels of features, thereby enhancing the distinguishability of features.

[0065] Traditional convolution operations treat each pixel in the HSI as a node, resulting in a significantly increased computational complexity of the adjacency matrix due to the excessive number of nodes. Therefore, in this study, the SLIC algorithm is adopted for superpixel segmentation, effectively reducing the number of nodes and generating three matrices, including the transformation matrix B ∈ R Nxr , the adjacency matrix A ∈ R rxr , and the superpixel information matrix S ∈ R rxk , where r represents the number of superpixels and k represents the number of bands after dimensionality reduction. Matrices B and A are as follows:

[0066]

[0067] Among them, Y pca represents the HSI after dimensionality reduction, and Flatten(·) represents the process of flattening Y pca along its spatial dimension. represents the i-th pixel in the flattened matrix, η(r j ) represents the set of elements of the i-th superpixel, and A(i, j) represents the value at the coordinate (r i , r j ) in the adjacency relationship matrix.

[0068] The adjacency weight matrix A1 of the first-order dynamic multi-hop graph can be constructed by utilizing the weight information between superpixel nodes, which is calculated using the Gaussian kernel function. The formula is as follows:

[0069]

[0070] The set ψ(r i ) represents the spatially adjacent superpixels of the superpixel ri, ε is a hyperparameter, and W(i, j) represents the weight value of the superpixel at the coordinate (r i , r j ). The depth-first search (DFS) algorithm is applied to each superpixel node to identify and record all paths that are z jumps away from the selected central node z. Then, the endpoints of these paths are labeled as r z . Therefore, the adjacency matrix for the z-th hop can be defined as:

[0071]

[0072] Among them, r1, r2,,,, r z represent the intermediate nodes of the path. Therefore, the adjacency feature matrix for the z-th hop is obtained by summing and averaging z - 1 adjacency feature matrices. It contains all the feature information from paths 1 - z. Based on this method, a series of k-hop graph adjacency matrices corresponding to different hop numbers can be generated, denoted as A1, A2,,,, A zThese matrices provide the basic support for the dynamic multi-hop graph structure. In this structure, the direct neighbors of the central superpixel node are regarded as the one-hop graph, the neighbors of its neighbors are regarded as the two-hop graph, and so on. This construction method effectively avoids redundancy while expanding the receptive field of a single node, enabling it to connect with more nodes across multiple hops, thereby enhancing the global perception ability of the network.

[0073] G1 = BN(WA(BN(S)+GCNB(BN(S),A1))) (5)

[0074] G2 = BN(WA(G1+GCNB(G1,A2))) (6)

[0075] G3 = BN(WA(G2+GCNB(G2,A3))) (7)

[0076] Where G i (i = 1, 2, 3) represents the feature matrix after i-hop graph convolution. WA effectively learns the weights of two input features by generating a two-dimensional vector and calculates their weighted sum. GCNB consists of Figure 2 the 7 sub-modules shown in (c).

[0077] Next, the present invention further combines the feature matrices obtained at each jump point and concatenates them along the channel dimension into G out , outputting a total of 192 channels for subsequent modules to use,

[0078] G out = Concat(G1,G2,G3) (8)

[0079] 2) Cross-Head Interactive Attention (CHIA). CHIA incorporates cross-attention into the multi-head attention mechanism to promote the interaction of feature information between different attention heads. This enhanced feature representation mechanism enables CHIA to further learn multi-hop features from DMPGC, promotes information sharing and reuse at different hop levels, and effectively reduces redundancy.

[0080] The input features of CHIA are initially divided into multiple heads, and then features are performed within each sub-head. The schematic diagram of the CHIA model is as Figure 3 shown.

[0081] To enable the model to more flexibly capture and utilize the rich information in the input data, for a given input, three feature matrices are generated through linear transformation: Q (Query), K (Key), and V (Value).

[0082] Q = G out W Q , K = G out W K, V = G out W V (9)

[0083]

[0084] where W Q , W K , W V are weight matrices, and their internal parameters are automatically learned through model training. Next, Q, K, and V are transformed into the multi-head form of four-dimensional tensors. For each attention head, the product of the query vector and the key vector is calculated, and then the attention weight matrix is obtained through the Softmax layer, and a dropout layer is added to prevent overfitting. Then, the attention weight matrix and the value vector are multiplied matrix-wise to capture the dependencies and interactions between positions in the input sequence. Finally, the outputs of each head are fused through cross-attention to achieve information sharing and reuse between heads. After linear transformation, summation and layer normalization are performed with some channels of the original input features, and finally all attention heads are concatenated along the number of channels.

[0085]

[0086] Equation (9) represents the attention calculation formula for the i-th head, where i = (1, 2..s). In Equation (10), s represents the number of heads in the multi-head attention. LayerNorm improves the training stability and effectively solves the problem of gradient vanishing or explosion. The head interaction layer (HIL) constructed using cross-attention promotes efficient information exchange between attention heads to enhance the feature representation.

[0087] Finally, the fused long-range spatial features are transformed back into the pixel form using the pixel transformation matrix B to produce an output of 50 channels, facilitating feature fusion in subsequent modules.

[0088] B. Composite Multi-Scale Convolutional Spatial-Spectral Attention Module (CMS2AM)

[0089] Traditional multi-scale convolution methods apply convolution operations with different kernel sizes on the same input features to extract information at different scales. However, directly using CNNs with different kernel sizes when processing HSI with a large spectral dimension leads to excessive parameters, resulting in overfitting and reduced unmixing performance. To address these challenges and enable deeper feature extraction, the present invention designs the CMS2AM. This module consists of two sub-modules: the Composite Multi-Scale Convolutional Residual Module (CMSCR) and the Dual Convolutional Spatial-Spectral Attention (DCS2A) module.

[0090] 1) Composite Multi-Scale Convolutional Residual Module (CMSCR).

[0091] In the CMSCR, this patent designs a composite convolutional network block: CNNB for multi-scale feature extraction, and its specific structure is as shown in Figure 2 (b). Initially, the input data passes through BN to improve stability and prevent overfitting, thereby accelerating network training. This module combines 1x1 convolution with kxk convolution and applies the LReLU activation function after convolution. The use of 1x1 convolution kernels can effectively reduce the number of parameters, mitigate overfitting, and at the same time retain sufficient feature information. In addition, when combined with kxk convolution kernels, the network can extract feature information of various scales to enhance its performance.

[0092] Next, HSI is convolved with CNNB modules of sizes 7x7, 5x5, and 3x3 respectively. To ensure the effective fusion of features at different scales, the present invention adds the three obtained feature matrices, and the formula is as follows:

[0093] F1 = CNNB_3(Y) + CNNB_5(Y) + CNNB_7(Y) (13)

[0094] where CNNB_3, CNNB_5, and CNNB_7 represent CNNBs using 3x3, 5x5, and 7x7 dimensions respectively. Repeat this operation three times to fully extract features, as shown in formulas (12) and (13). Moreover, to improve the stability of the model, the present invention concatenates the five channels of the original image with the above three output feature matrices to form C out , and uses 1x1 convolution to adjust the output to 256 channels for subsequent modules, and the specific operation is as follows:

[0095] F2 = CNNB_3(F1) + CNNB_5(F1) + CNNB_7(F1) (14)

[0096] C out = Conv 1×1 (Concat(CNNB_3(F2) + CNNB_5(F2) + CNNB_7(F2) + Y)) (15)

[0097] 2) Dual-convolution spatial-spectral attention module (DCS2A).

[0098] DCS2A consists of two parts: a spectral attention module and a spatial attention module. Each module further promotes feature extraction through double-layer convolution, thereby achieving a more comprehensive fusion of feature information. For the input multi-scale feature image, DCS2A can further enhance the correlation between features at different scales. In addition, the computational complexity of DCS2A is relatively low and is particularly efficient when processing a large number of multi-scale feature maps. The schematic diagram of the DCS2A model is as shown in Figure 4 .

[0099] The spectral attention mechanism aims to extract key information from the feature map using the channel dimension. Specifically, first, for the input feature map C out global average pooling and max pooling are applied to capture the comprehensive spectral information in the spatial dimension. These operations are defined as follows:

[0100] R avg = AvgPool(C out ), R max = MaxPool(C out ) (16)

[0101] R avg and R max are of shape [Be, Ce, 1, 1], where Be represents the batch size and Ce represents the number of channels. Subsequently, 1x1 convolution operations are performed on R avg and R max respectively, with a kernel size of 1x1. The generated feature maps are processed through the ReLU activation function, and the above operations are repeated twice, as follows:

[0102] R' avg = ReLU(Conv 1×1 (ReLU(Conv 1×1 (R avg )))),

[0103] R' max = ReLU(Conv 1×1 (ReLU(Conv 1×1 (R max )))) (17)

[0104] Finally, the spectral attention map Aspectral is obtained by element-wise addition, as follows:

[0105] A spectral = R' avg + R' max (18)

[0106] The spatial attention design aims to further extract the key information in the spatial dimension from the feature map. First, for each spatial position, average pooling and max pooling operations are performed on the input feature map in the channel dimension:

[0107] T avg = AvgPool(A spectral , dim = 1), T max = MaxPool(A spectral , dim = 1) (19)

[0108] In the above formula, the operation generates a feature map with the shape of

Be, 1, He, We

[0109] T Concat = Concat([T avg , T max , dim = 1) (20)

[0110] To capture a wider range of regional features, the present invention performs two 7x7 convolution operations on the cascaded feature map as follows:

[0111] A spatial = Conv 7×7 (ReLU(Conv 7×7 (T Concat ))) (21)

[0112] After obtaining the final spatial attention features, the present invention combines them with the input feature map through element-wise vector multiplication and addition.

[0113] CNN out = C out ⊙ A spatial + C out (21)

[0114] By integrating spectral and spatial information, the feature representation ability of DCS2A is further enhanced. This dual attention mechanism performs excellently in mixed tasks, can more effectively handle spectral and spatial variations in complex scenes, thereby improving the accuracy of mixing.

[0115] C. Loss Function

[0116] The present invention adopts the mean squared error (MSE) as the loss function. MSE is a widely used metric in regression tasks, aiming to minimize the difference between the reconstructed image and the original image. The formula of MSE is as follows, y i and represent the original and estimated spectral vectors respectively,

[0117]

[0118] HSI is often affected by various types of noise. Although some algorithms can reduce image noise, they often sacrifice edge and detail information. To solve this problem, the present invention proposes a new loss function: an adaptive weight variational loss function based on the Sobel edge operator and Gaussian function (SGATV), aiming to effectively reduce noise while maximizing the retention of edge and detail information. The proposed loss function first uses the Sobel edge detection operator to calculate pixel gradients and extract edge information. Then, an adaptive weighting mechanism is introduced through the Gaussian function, which dynamically adjusts pixel weights according to the local characteristics of pixels. The gradients of the Sobel edge operator in the horizontal (x) and vertical (y) directions can be expressed as:

[0119]

[0120] The image gradient can be obtained through convolution operation, as shown below:

[0121]

[0122] The gradients calculated using the Sobel operator are incorporated into the Gaussian function to construct the adaptive weight matrix w.

[0123]

[0124] To mitigate the impact of noise and retain important features in the data, the present invention adopts the TV loss, which has the following forms in the vertical and horizontal directions, where nl and nc represent the length and width of HSI, respectively.

[0125]

[0126] The final SGATV loss formula is as follows:

[0127]

[0128] SGATV effectively reduces noise through its adaptive weighting mechanism and retains edge information. Finally, the total loss of this model is shown as follows:

[0129] LOSS = Loss MSE + Loss SGATV (28)

[0130] 3. Technical effects

[0131] To evaluate the effectiveness of this model, the present invention selected two synthetic datasets and four real datasets for algorithm experiments to verify the effectiveness. The evaluation of each synthetic dataset was carried out at three additive signal-to-noise ratios (SNRs), and the tested SNRs were 20 dB, 30 dB, and 40 dB respectively. For the evaluation metrics, on the synthetic datasets, since the true abundances were known, the error between the estimated abundances and the true abundances was quantified by the signal reconstruction error (SRE). The formula is shown as follows. However, for some real HSIs, due to the lack of accurate abundance information, only the result images are presented.

[0132]

[0133] The number of iterations of this model was set to 1500 times on both synthetic and real datasets. In addition, this method utilized the Adam optimizer with the learning rate set to 1e-3. Python 3.8.19 was used as the programming language for the experiments, and PyTorch 1.10.1 together with CUDA version 11.3 was used as the programming environment. The experiments were conducted on a computer equipped with an Intel Core i5-12600KF CPU, 32 GB of RAM, and an NVIDIA GeForce RTX 3060 (12 GB GPU memory). The selection of superpixels and the number of superpixels in SGATV were flexibly adjusted according to the changes in the datasets.

[0134] The endmember curves and images of the two synthetic datasets are as Figure 5 shown, and the true color images of the four real datasets are as Figure 6 shown.

[0135] 1) Synthetic dataset 1 (DC1): The DC1 dataset used a linear mixing model (LMM) with 5 endmembers to generate a 75x75 pixel image. Among them, the first row of DC1 consisted of 5 pure pixels, and the second to fifth rows each contained mixed pixels of 2 to 5 land covers, with the background being a mixture of all land covers. The endmembers were selected from the USGS spectral library, and the abundance map consisted of 5 spatially uniform square regions of endmember spectra, selected according to the minimum spectral angle of 4.44 in the USGS spectral library, generating a total of 240 spectral features.

[0136] The SRE values of the model of the present invention on this dataset at three signal-to-noise ratios (20 dB, 30 dB, 40 dB) were 7.99, 11.65, and 15.86 respectively, and the generated abundance images are as Figure 7 shown. By comparing with the true abundance image (GT), it can be found that the PMGMCN algorithm performs stably at different signal-to-noise ratios and can obtain good unmixing results under various conditions. This highlights the advantages of this model in fully considering the spatial distance features and fusing them, while combining the adaptive total variation loss.

[0137] 2) Synthetic dataset 2 (DC2): The DC2 dataset generates a 100x100 pixel image through an LMM with 9 endmembers. The endmember spectral library is the same as that of the DC1 dataset, and the abundance map is sampled from a Dirichlet distribution centered on a Gaussian random field, forming a patchy smooth image with steep transitions.

[0138] The SRE values of the model of the present invention on this dataset at three signal-to-noise ratios (20 dB, 30 dB, 40 dB) are respectively: 9.35, 11.29, and 14.32. The generated abundance images are as Figure 8 shown. By comparing with the true abundance image (GT), it can be found that PMGMCN can unmix the correct ground object distribution in each signal-to-noise ratio environment. It can make full use of spatial context information and graph structure information, thus achieving a good unmixing effect.

[0139] 3) Jasper Ridge: The dataset was collected by an AVIRIS sensor and contains a 100x100 pixel image. After removing the low-quality bands, 198 channels are used for unmixing, including four main types of endmembers: trees, waterbodies, soil, and roads.

[0140] The abundance image of PMGMCN on this dataset is as Figure 9 shown. By comparing with the true color image of this dataset, it can be found that the method of the present invention can accurately capture the main distribution areas of the four ground objects. The details shown in the abundance map are more abundant, and the sharpness of the boundaries and the degree of detail retention are significantly superior. Generally speaking, the method of the present invention shows significant advantages in the unmixing task of the Jasper Ridge dataset. The method of the present invention can more effectively suppress background noise, maintain the clarity and details of the ground object boundaries, and the generated abundance map has higher contrast and clarity, fully demonstrating its superior performance in complex scenes.

[0141] 4) Samson: The dataset is provided by Oregon State University and is obtained through a pushbroom visible in the near-infrared sensor. The dataset consists of a 95x95 grid of pixels, and each pixel contains 156 channels. The image represents three endmembers of materials: soil, trees, and water.

[0142] The abundance image of PMGMCN on this dataset is as Figure 10As shown, by comparing with the true-color image of this dataset, it can be found that: the algorithm proposed by the present invention performs excellently in the abundance estimation of three types of ground objects, namely Soil, Tree, and Water. In particular, it shows significant advantages in capturing edge details and suppressing noise, which fully verifies the superiority of the model of the present invention in comprehensively considering spatial and spectral information. Moreover, by combining superpixel features and pixel features, this method obtains better abundance estimation.

[0143] 5) Cuprite: The Cuprite dataset was collected using the AVIRIS sensor in the Cuprite area of Nevada and contains an image of 250x191 pixels. After removing the noise and water absorption channels, 188 bands were retained for unmixing. It contains 12 unique mineral endmembers, including alunite, andradite, buddingtonite, dumortierite, kaolinite_1, kaolinite_2, muscovite, montmorillonite, nontronite, pyrope, sphene, and chalcedony.

[0144] The abundance image of PMGMCN on this dataset is as Figure 11 As shown, by comparing with the true-color image of this dataset, it can be found that: the method of the present invention performs excellently in maintaining edge details and reducing the influence of noise. The contrast and clarity of the abundance map are significantly improved, demonstrating excellent unmixing ability. This benefits from the advantage that the model of the present invention expands the receptive field and captures more refined spectral information.

[0145] 6) Urban: The dataset was obtained by the HY-DICE sensor and contains an image of 307x307 pixels. After removing the low-quality channels, 162 bands were retained for unmixing, and there are four main endmembers: tree, water, soil, and road.

[0146] The abundance image of PMGMCN on this dataset is as Figure 12 As shown, by comparing with the true-color image of this dataset, it can be found that: whether it is in retaining the edge details of complex ground objects or in accurately distinguishing between different ground object types, the method of the present invention shows excellent performance, providing a more accurate and reliable solution for hyperspectral image unmixing.

[0147] The above are only the embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. All equivalent structural or equivalent process transformations made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, are similarly included in the patent protection scope of the present invention.

Claims

1. A parallel dynamic multi-hop graph and compound multi-scale convolutional hyperspectral sparse unmixing network, characterized in that It includes a Dynamic Multi-hop Graph Interaction Attention Module (DMGIAM) and a Composite Multi-scale Convolutional Spatial-Spectral Attention Module (CMS2AM); Among them, the Dynamic Multi-hop Graph Interaction Attention Module (DMGIAM) is used to extract long-distance spatial dependence features of hyperspectral images, and includes two sub-modules: Dynamic Multi-hop Perception Graph Convolution (DMPGC) and Cross-Head Interaction Attention (CHIA); The Dynamic Multi-hop Perception Graph Convolution (DMPGC) generates an adjacency matrix through superpixel segmentation and adaptively weights and aggregates graph features of different hop numbers; The Cross-Head Interaction Attention (CHIA) fuses multi-hop features through a multi-head attention mechanism and a cross-head interaction layer; Among them, the Composite Multi-scale Convolutional Spatial-Spectral Attention Module (CMS2AM) is used to extract local multi-scale spectral-spatial features, and includes a Composite Multi-scale Convolutional Residual Module (CMSCR) and a Dual Convolutional Spatial-Spectral Attention Module (DCS2A); The Composite Multi-scale Convolutional Residual Module (CMSCR) extracts local features of different scales through multi-scale convolutional kernels and performs residual fusion; The Dual Convolutional Spatial-Spectral Attention Module (DCS2A) enhances feature representation through spectral attention and spatial attention mechanisms; The network structure includes an Adaptive Weighted Total Variation Loss Function (SGATV), which is used to dynamically adjust weights by combining the Sobel edge detection operator and the Gaussian function, promoting the smoothing and denoising of the abundance matrix; the parallel structure of the DMGIAM and CMS2AM inputs the dimensionality-reduced hyperspectral data and the original data respectively, and outputs the abundance matrix through feature splicing and fusion.

2. The parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network according to claim 1, wherein The Dynamic Multi-hop Perception Graph Convolution (DMPGC) specifically includes: Using SLIC superpixel segmentation to generate a pixel-superpixel transformation matrix B, an adjacency matrix A, and a superpixel information matrix S; Generating a multi-hop adjacency matrix through the Depth-First Search (DFS) algorithm and calculating weights using a Gaussian kernel function; Adopting a Weighted Aggregation (WA) module to dynamically adjust the weights of features of different hop numbers and generate fused multi-hop graph features.

3. The parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network according to claim 1, wherein The Cross-Head Interaction Attention (CHIA) specifically includes: Dividing the input features into multiple attention heads and generating query Q, key K, and value V matrices respectively; Implementing feature interaction and information sharing between different attention heads through a cross-attention mechanism; Adopting Layer Normalization (LayerNorm) and residual connections to improve training stability.

4. The parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network according to claim 1, characterized in that The Composite Multi-scale Convolutional Residual (CMSCR) specifically includes: Using a 1×1 convolutional kernel to reduce the parameter dimension and parallelly extracting multi-scale features with 3×3, 5×5, and 7×5 convolutional kernels; Adding the multi-scale features and splicing them with the original input, and adjusting the number of output channels through a 1×1 convolution.

5. The parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network according to claim 1, characterized in that, The Dual Convolutional Spatial-Spectral Attention (DCS2A) specifically includes: Spectral attention module: Generating channel weights through global average pooling and max pooling to enhance the features of key spectral channels; Spatial attention module: Generating spatial weights through channel-dimensional pooling and cascaded convolutions to enhance the features of key spatial positions; Fusing the spectral and spatial attention weights with the input features to generate an enhanced feature map.

6. The parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network according to claim 1, characterized in that, The construction of the Adaptive Weighted Total Variation Loss Function (SGATV) includes: Calculate the pixel gradient using the Sobel operator and generate an adaptive weight matrix by combining with the Gaussian function; Combine the weight matrix with the total variation TV loss to retain edge details while smoothing the noise.

7. The parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network according to claim 1, wherein The network adopts a parallel structure, splices the long-distance features output by DMGIAM and the multi-scale local features output by CMS2AM, and outputs the abundance matrix after fusion through two layers of CBLM modules.

8. The parallel dynamic multi-hop graph and composite multi-scale convolutional hyperspectral sparse unmixing network according to claim 1, characterized in that, The network is trained by the Adam optimizer, the learning rate is set to le-3, and the total loss function is the weighted sum of the mean square error MSE and the SGATV loss.

Images