Hyperspectral image classification method and system based on spectrum-space double-branch token feature fusion network

Abstract

The invention discloses a hyperspectral image classification method and system based on a spectrum-space double-branch token feature fusion network, and the method comprises the steps: obtaining and processing an original hyperspectral image, carrying out the feature extraction of the processed image, and obtaining a three-dimensional output feature; global average pooling and flattening are carried out on the three-dimensional output features along the spectrum dimension, obtained two-dimensional features and local space spectrum details are spliced, and complete space and spectrum features are constructed; enhancing and optimizing the key spatial features and the spectral features by using global grouping coordinate attention and gating bottleneck channel attention to obtain spatial features and spectral features with discrimination ability; converting the spatial features and the spectral features with discrimination power into a sequence with a fixed length to obtain a spatial feature mark and a spectral feature mark, and fusing the spatial feature mark and the spectral feature mark by using a gating enhancement token mechanism to obtain an enhanced spatial mark and an enhanced spectral mark; and generating an image classification result based on the enhanced spatial mark and the enhanced spectral mark.

Description

Technical Field

[0001] This invention belongs to the field of hyperspectral image classification technology, specifically relating to a hyperspectral image classification method and system based on a spectral-spatial dual-branch token feature fusion network. Background Technology

[0002] Hyperspectral images acquired through narrowband imaging typically contain tens to hundreds of continuous spectral bands, capturing near-continuous spectral response curves of land surface features. This unique data characteristic endows HSI with rich spectral discrimination information, enabling it to enhance the differentiation of land cover types through subtle spectral differences. During acquisition, hyperspectral images simultaneously record the spatial geometric coordinates of each pixel, ultimately forming a three-dimensional data cube that integrates two-dimensional spatial structure and one-dimensional spectral features. As a core component of hyperspectral remote sensing applications, hyperspectral image classification has become a research hotspot, with its main goal being to accurately assign land cover labels to individual pixels in the image. Notably, this technology demonstrates irreplaceable application value in multiple fields, including environmental quality monitoring, regional transportation planning, crop yield estimation, land use classification, and improving the accuracy of engineering surveying.

[0003] In recent years, deep learning has become an extremely successful technology, demonstrating outstanding performance in the field of computer vision. Therefore, deep learning technology has been introduced into the field of hyperspectral image classification for remote sensing imagery. Compared with traditional methods that rely on manually designed features, deep learning can automatically learn high-level abstract features from complex hyperspectral data. Early research focused on the spectral characteristics of images, including Support Vector Machines (SVM), Random Forests (RF), Principal Component Analysis (PCA), and Logistic Regression (LR). However, due to the typically low spatial resolution of hyperspectral images, pixel mixing, and the presence of different objects within the same spectrum or vice versa, traditional methods often struggle to capture detailed information, resulting in poor classification accuracy. Furthermore, these methods also have other limitations, including a high dependence on prior knowledge, insufficient image feature representation capabilities, and weak generalization ability.

[0004] Hyperspectral images consist of multiple consecutive spectral bands, each corresponding to a specific wavelength range and capturing the unique spectral reflectance characteristics of land surface features. Different land cover types exhibit significant reflectance differences across wavelengths, and certain land features show enhanced reflectance characteristics in specific bands. By selecting three bands from the band set to synthesize an RGB image, hyperspectral data can be converted into a visually interpretable format, aiding in land cover spectral analysis. Notably, bands with adjacent wavelengths exhibit high spectral redundancy, requiring targeted dimensionality reduction during feature extraction to improve classification efficiency. Traditional channel attention mechanisms focus on high-reflectance, information-rich bands while neglecting low-reflectance bands, which affects the classification performance of cover types dependent on the latter and reduces average precision (AA). Furthermore, traditional methods compensate for spatial information loss by stacking feature channels, but this introduces redundant parameters, increases model complexity, and leads to low channel feature utilization and blurred boundaries of small targets, due to insufficient modeling of inter-channel dependencies. Summary of the Invention

[0005] This invention aims to address the shortcomings of existing technologies and provides the following solutions: A hyperspectral image classification method based on a spectral-spatial dual-branch token feature fusion network includes the following steps: The original hyperspectral image is acquired, preprocessed, and then combined spectral-spatial feature extraction is performed on the preprocessed image to obtain three-dimensional output features. The three-dimensional output features are globally averaged along the spectral dimension to obtain two-dimensional features containing global contextual information. At the same time, the three-dimensional output features are flattened in the channel and spectral dimensions to preserve local spatial spectral details. The two-dimensional features and the local spatial spectral details are then concatenated in the channel dimension to construct complete spatial and spectral features that integrate local details and global semantics. By using global grouped coordinate attention to enhance key spatial features in the spatial and spectral features, and by using gated bottleneck channel attention to optimize spectral features in the spatial and spectral features, discriminative spatial features and discriminative spectral features are obtained. The discriminative spatial features and discriminative spectral features are converted into fixed-length sequences using learnable projection to obtain spatial feature tags and spectral feature tags. The spatial feature tags and spectral feature tags are then fused using a gated enhancement token mechanism to obtain enhanced spatial tags and enhanced spectral tags. Image classification results are generated based on the enhanced spatial label and the enhanced spectral label.

[0006] Preferred methods for obtaining three-dimensional output features include: Principal component analysis was used to reduce the dimensionality of the original hyperspectral image to obtain the dimensionality-reduced image data. The dimensionality-reduced image data is preliminarily extracted using three-dimensional convolution to obtain initial spectral features; The initial spectral features are pooled and stitched using a spectral band enhancer to obtain the three-dimensional output features.

[0007] Preferably, the method for obtaining the discriminative spatial features includes: The spatial and spectral features are divided into several groups, and global average pooling is performed on the feature map of each group in the height dimension and global max pooling in the width dimension to obtain the pooled feature map. Convolution operations are performed on the pooled feature maps respectively to extract local features and obtain the output feature map; The height-direction attention weight and width-direction attention weight are calculated based on the output feature map, and the discriminative spatial features are obtained based on the spatial and spectral features, the height-direction attention weight, and the width-direction attention weight.

[0008] Preferably, the method for obtaining the discriminative spectral features includes: The spatial and spectral features are processed using three parallel bottleneck structures to obtain channel features; The attention score and gating score are calculated based on the channel features, and the spectral-spatial joint features are obtained based on the attention score and the gating score. The spatial and spectral features and the spectral-spatial joint features are fused and the channel dimensions are normalized to obtain the discriminative spectral features.

[0009] The present invention also provides a hyperspectral image classification system based on a spectral-spatial dual-branch token feature fusion network. The system applies the above-mentioned method and includes: an image preprocessing module, a feature fusion module, a feature extraction module, a feature fusion module, and a result generation module. The image preprocessing module is used to acquire the original hyperspectral image, preprocess the original hyperspectral image, and perform joint spectral-spatial feature extraction on the preprocessed image to obtain three-dimensional output features; The feature fusion module is used to perform global average pooling on the three-dimensional output features along the spectral dimension to obtain two-dimensional features containing global context information, and simultaneously flatten the three-dimensional output features in the channel and spectral dimensions to preserve local spatial spectral details. Then, the two-dimensional features and the local spatial spectral details are concatenated in the channel dimension to construct complete spatial and spectral features that fuse local details and global semantics. The feature extraction module enhances key spatial features in the spatial and spectral features by using global grouped coordinate attention, and optimizes spectral features in the spatial and spectral features by using gated bottleneck channel attention, thereby obtaining discriminative spatial features and discriminative spectral features. The feature fusion module uses learnable projection to convert the discriminative spatial features and the discriminative spectral features into fixed-length sequences, respectively, to obtain spatial feature labels and spectral feature labels. Then, it uses a gated enhancement token mechanism to fuse the spatial feature labels and the spectral feature labels to obtain enhanced spatial labels and enhanced spectral labels. The result generation module generates image classification results based on the enhanced spatial label and the enhanced spectral label.

[0010] Preferably, the workflow of the image preprocessing module includes: Principal component analysis was used to reduce the dimensionality of the original hyperspectral image to obtain the dimensionality-reduced image data. The dimensionality-reduced image data is preliminarily extracted using three-dimensional convolution to obtain initial spectral features; The initial spectral features are pooled and stitched using a spectral band enhancer to obtain the three-dimensional output features.

[0011] Preferably, the process of obtaining the discriminative spatial features in the feature extraction module includes: The spatial and spectral features are divided into several groups, and global average pooling is performed on the feature map of each group in the height dimension and global max pooling in the width dimension to obtain the pooled feature map. Convolution operations are performed on the pooled feature maps respectively to extract local features and obtain the output feature map; The height-direction attention weight and width-direction attention weight are calculated based on the output feature map, and the discriminative spatial features are obtained based on the spatial and spectral features, the height-direction attention weight, and the width-direction attention weight.

[0012] Preferably, the process of obtaining the discriminative spectral features in the feature extraction module includes: The spatial and spectral features are processed using three parallel bottleneck structures to obtain channel features; The attention score and gating score are calculated based on the channel features, and the spectral-spatial joint features are obtained based on the attention score and the gating score. The spatial and spectral features and the spectral-spatial joint features are fused and the channel dimensions are normalized to obtain the discriminative spectral features.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention uses "divide and conquer enhancement - unified fusion - collaborative decision-making" as its core logic. It optimizes initial features through cascaded 3D convolution and spectral enhancers, constructs a dual-branch structure of gated bottleneck channel attention and global group coordinate attention to strengthen spectral dependence and global spatial correlation, respectively, and uses a tokenization mechanism to compress standardized features. It also utilizes a gated enhanced token attention module to achieve deep interaction of spectral and spatial labels to solve the problems of "different objects with the same spectrum" and "same objects with different spectra". Finally, it balances the contributions of the two branches through dual-path classification and multi-strategy integrated decision-making logic to improve classification stability in complex scenarios. This invention provides an innovative solution for hyperspectral classification that balances accuracy and efficiency. Attached Figure Description

[0014] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the network process according to an embodiment of the present invention; Figure 3 This is a schematic diagram of global grouped coordinate attention in an embodiment of the present invention; Figure 4 This is a schematic diagram of the attention required for the gated bottleneck channel in an embodiment of the present invention. Figure 5 This is a schematic diagram of the gated enhanced token mechanism according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a spectral band enhancer according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a token feature extractor according to an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Example 1 In this embodiment, as Figure 1 , Figure 2 As shown, a hyperspectral image classification method based on a spectral-spatial dual-branch token feature fusion network includes the following steps: S1. Acquire the original hyperspectral image, preprocess the original hyperspectral image, and perform joint spectral-spatial feature extraction on the preprocessed image to obtain three-dimensional output features.

[0019] The methods for obtaining three-dimensional output features include: using principal component analysis to reduce the dimensionality of the original hyperspectral image to obtain dimensionality-reduced image data; using three-dimensional convolution to perform preliminary spatial and spectral feature extraction on the dimensionality-reduced image data to obtain initial spectral features; and using a spectral band enhancer to perform pooling and stitching processing on the initial spectral features to obtain three-dimensional output features.

[0020] In this embodiment, hyperspectral images (HSIs) contain hundreds of continuous spectral bands, providing rich information for land cover classification. However, they have inherent drawbacks such as high data redundancy, low signal-to-noise ratio, and high computational complexity in model processing. In this embodiment, dimensionality reduction is achieved through principal component analysis (PCA), combined with spectral band enhancer (SBE) for feature optimization, thereby constructing a synergistic mechanism of dimensionality compression and feature optimization. SBE not only solves the inherent limitations of hyperspectral data but also provides a high-quality input foundation for subsequent feature extraction modules.

[0021] PCA, as the first stage of preprocessing, aims to reduce data dimensionality and redundancy. Strong correlations exist between bands in the original hyperspectral images, which not only increases computational burden but can also lead to model overfitting. Given a hyperspectral data cube, it can be represented as... In this framework, X represents the original hyperspectral image for the entire classification system, while W, H, and B correspond to the width, height, and number of spectral bands of the hyperspectral image, respectively. Each pixel in X is associated with a one-hot encoded label vector. Where C represents the number of land cover categories. Principal component analysis (PCA) is then applied along the spectral dimension of the original hyperspectral image data X. After dimensionality reduction, the spatial resolution of the data remains unchanged, while the number of spectral bands decreases from B to b, ultimately resulting in the transformed data cube. This process enables PCA to effectively eliminate redundant spectral information inherent in hyperspectral image data while preserving spatial details. PCA projects high-dimensional spectral data into a low-dimensional orthogonal subspace through linear transformation, eliminating noise and redundancy while retaining key features. This reduces the amount of data and enhances inter-band independence through orthogonalization, providing accurate input for subsequent spectral feature learning. However, PCA only performs linear transformation by maximizing global variance, failing to capture the task-related correlation between spectral bands and land cover categories.

[0022] To overcome this limitation, the data after dimensionality reduction via principal component analysis is first input into a 3D convolution module for preliminary spatial and spectral feature extraction. This 3D convolution operation uses a 3D convolution kernel of size to perform sliding calculations along the spatial and spectral dimensions, effectively capturing correlated spectral and spatial features within local regions. This process not only enhances feature discriminability but also lays a rich feature foundation for subsequent spectral optimization through multi-channel mapping.

[0023] Spectral band enhancer (SBE) is a post-processing step following principal component analysis (PCA), focusing on spectral enhancement features after PCA, such as... Figure 6 As shown. Although PCA dimensionality reduction eliminates some redundant information between bands, the contribution of different bands to land cover classification varies significantly. SBE achieves precise enhancement through a dual-pooling structure and a multilayer perceptron: First, the initial spectral features extracted by 3D convolution are processed by global average pooling and max pooling to compress the spatial dimension and condense the spectral information; then, the two sets of pooled features are concatenated, and a multilayer perceptron consisting of two 3D convolutional layers and a Dropout layer is used to adaptively learn the discrimination weights of each band based on the concatenated dual-pooling features. Then, the weights are normalized to the [0,1] interval by the Sigmoid activation function to generate interpretable importance coefficients to quantify the contribution of each band to land cover classification; finally, the bands are differentially weighted to strengthen the contribution of key bands while suppressing noise or interference in secondary bands.

[0024] These three components form a progressive preprocessing chain, sequentially achieving dimensionality reduction, preliminary fusion, and accuracy optimization. Principal component analysis performs simple filtering of spectral dimensions through linear transformation, 3D convolution achieves preliminary fusion and abstraction of local spatial-spectral features, and the spectral band enhancer fine-tunes the spectral dimensions according to task requirements, ultimately yielding 3D output features. This collaborative design not only solves the spectral redundancy problem in hyperspectral data but also effectively enhances the key spectral-spatial features for classification through a data-driven approach. This preprocessing chain provides high-quality input for subsequent modules such as global grouping coordinate attention and gated bottleneck channel attention, laying a solid foundation for feature learning throughout the network and further improving the final classification performance.

[0025] S2. Perform global average pooling on the spectral dimension of the 3D output features to obtain 2D features containing global contextual information. At the same time, flatten the 3D output features on the channel and spectral dimensions to preserve local spatial spectral details. Then, concatenate the 2D features and local spatial spectral details on the channel dimension to construct complete spatial and spectral features that integrate local details and global semantics.

[0026] S3. Use global grouped coordinate attention to enhance key spatial features in spatial and spectral features, and use gated bottleneck channel attention to optimize spectral features in spatial and spectral features to obtain discriminative spatial features and discriminative spectral features.

[0027] The method for obtaining discriminative spatial features includes: dividing spatial and spectral features into several groups, and performing global average pooling and global max pooling on the feature maps of each group to obtain pooled feature maps; performing convolution operations on the pooled feature maps to extract local features to obtain output feature maps; calculating height-direction attention weights and width-direction attention weights based on the output feature maps, and obtaining discriminative spatial features based on spatial and spectral features, height-direction attention weights, and width-direction attention weights.

[0028] In this embodiment, hyperspectral images not only contain rich spectral information but also crucial spatial information, which is essential for distinguishing different land cover types. Traditional convolutional neural networks, when processing complex visual tasks, cannot simultaneously capture global information in both height and width dimensions, resulting in limited spatial-spectral feature representation capabilities. The global grouped coordinate attention module dynamically captures global information in both height and width directions through global pooling operations and attention mechanisms. This module enhances important spatial features while suppressing irrelevant information by generating spatial coordinate attention weights. Figure 3 , Figure 4 As shown, its core innovation lies in the global grouping mechanism: the feature maps are first grouped by channel, and then attention weights for spatial coordinates (height and width directions) are learned for each group. This design maintains the sensitivity of coordinate attention to spatial location, and reduces redundant interference between channels through grouping, making it more suitable for processing high-dimensional spatial-spectral features generated after hyperspectral feature extraction. Global Grouped Coordinate Attention (GGCA) Figure 3 As shown, the mechanism divides the channels into multiple groups. Each group independently learns local features while capturing global information through a cross-dimensional interaction module shared between groups. First, the input feature map is... The input feature map is divided into G groups from the number of channels C, with each group containing C / G channels. Here, B represents the batch size, and H and W represent the height and width of the feature map, respectively. The grouped feature map is denoted as... Subsequently, the feature maps of each group are subjected to global average pooling in the height dimension and global max pooling in the width dimension. in, This indicates that global average pooling is performed along the height dimension. This indicates that global max pooling is performed along the height dimension. This indicates that global average pooling is performed along the width dimension. The expression `AvgPool` represents global max pooling in the width dimension, while `AvgPool` represents global average pooling. Global average pooling emphasizes regional trends, suppresses local noise, highlights common features, and mitigates interference from mixed pixels. In contrast, global max pooling focuses on salient features, accurately captures the core of the target, emphasizes local uniqueness, and prevents background feature occlusion. Combining the two enables complementary feature extraction, allowing the model to efficiently extract discriminative information from hyperspectral features. The global grouped coordinate attention module considers both the global distribution of the target and local details, resists noise interference, and enhances directional spatial features. Therefore, this model can achieve accurate target discrimination in complex hyperspectral scenes, thereby improving classification accuracy and robustness.

[0029] For each grouped feature map, a shared cross-dimensional interaction module is used for feature processing. This module generates more semantically meaningful feature representations by learning the non-linear relationships between channels. These representations are used to calculate attention weights, thereby enhancing important spatial features while suppressing unimportant ones, as shown in the following formula: In the above expression, Conv represents the convolution operator, which operates on the input feature map. , , and After performing average pooling (avg) or max pooling (max) in the height (h) or width (w) direction, a convolution operation is performed. This operation slides a preset convolution kernel across the input feature map, performs product summation on each local region, thereby extracting local features from the input feature map, realizing feature transformation or fusion, and generating the corresponding output feature map. , , and .

[0030] Subsequently, attention weights in the height and width directions are generated by summing the outputs of the convolutional layers and applying the sigmoid activation function. This also achieves the fusion of the results after pooling operations in each direction. Where σ represents the Sigmoid activation function, express…, express….

[0031] Finally, multiplying the input feature map by the attention weights yields the spatially sensitive attention weights: Here, O represents the attention weight feature map that is spatially sensitive; this process can dynamically enhance the effective region while suppressing noisy interference regions.

[0032] In this embodiment, the GGCA mechanism reduces channel interference through group learning, captures directional features using coordinate separation, and preserves spatial context using global pooling. This module processes high-channel features generated by 3D convolution, adapting to the high dimensionality, spatial heterogeneity, and detail sensitivity of hyperspectral images, thereby reducing redundancy and avoiding inter-channel interference. After SBE completes the key band selection, GGCA focuses on the spatially effective regions of these bands, forming a progressive optimization with SBE. While SBE locks onto sensitive bands, GGCA accurately locates the spatial distribution, achieving dual focus on bands and locations to improve classification accuracy. This module also maintains the width and height spatial dimensions unchanged, ensuring that the spatial correlation of target objects is passed to the spatial branch. GGCA not only plays a role in feature purification but also enhances the model's ability to learn spatial morphology, providing support for subsequent tasks.

[0033] The method for obtaining discriminative spectral features includes: processing spatial and spectral features using three parallel bottleneck structures to obtain channel features; calculating attention scores and gating scores based on channel features, and obtaining spectral-spatial joint features based on attention scores and gating scores; fusing spatial and spectral features and spectral-spatial joint features, and normalizing the channel dimensions to obtain discriminative spectral features.

[0034] In this embodiment, in the hyperspectral image classification task, the hyperspectral image contains hundreds of continuous spectral bands, which carry key information for distinguishing land cover features. However, atmospheric scattering, sensor noise, and band correlation introduce significant redundancy. Traditional convolutional neural networks process all bands equally, making it difficult to extract effective value from high-dimensional data. Therefore, this embodiment proposes a gated bottleneck channel attention mechanism (GBCA), such as... Figure 4As shown, this mechanism can automatically learn and enhance the weights of key classification bands based on hyperspectral channel information, thereby improving classification performance. This module has three major innovations: First, it adopts a bottleneck structure, reducing computational overhead while enhancing feature abstraction capabilities and strengthening cross-band fusion to adapt to the challenges of high-dimensional, small-sample hyperspectral data; second, it introduces a self-attention query-key-value mechanism, calculating spectral response correlation by multiplying the feature matrix through Fq and Fk branches, and obtaining channel importance scores via softmax to filter key bands; third, it uses a superimposed sigmoid gating mechanism to dynamically filter scores a second time, suppressing noisy bands. This dual-scoring mechanism balances global competition and local suppression, avoiding overactivation. In terms of feature integration, a concat mode is used to reveal deep spatial-spectral correlations through feature recombination and dimensionality reduction, adapting to complex mixed terrain regions. Finally, it combines BatchNorm and ReLU to enhance model expressive power while maintaining stable feature distribution. This mechanism can dynamically amplify key spectral features and suppress redundant features, improving the classification accuracy of complex land cover and alleviating overfitting problems in small-sample scenarios.

[0035] First, for the input feature map Three parallel bottleneck structures generate query (Q), key (K), and value (V) through their convolutional paths, thus producing channel features: in, Fq This indicates the relationship between different spectral bands, i.e., which band is more important. Fk This indicates the characteristic representation of the spectral bands, that is, what characteristics each band has. Fv ReLU represents the enhanced spectral information to be transmitted, i.e., the spectral features to be transmitted. ReLU represents a nonlinear activation function.

[0036] Next, the similarity between the query vector and the key vector is calculated using matrix multiplication, compressing spatial similarity information to the channel dimension. The similarity scores are summed along the spatial dimension to obtain the composite score for each channel, thereby aggregating the spectral band response features within the spatial region and determining the overall importance of each band. Subsequently, the channel dimension scores are subjected to Softmax normalization to highlight the relative importance of key channels, automatically distinguishing key spectral band combinations and generating attention weights, as shown in the following formula: in, f _ sim The spatial similarity matrix quantifies the correlation strength between different pixels in an image, and Matmul represents matrix multiplication. FkT express Fk transpose, f _ sumThe channel importance score represents the degree of channel importance between different spatial locations. h The height dimension starts from 1. w This indicates that the width dimension starts from 1. xi Indicates the first i The original scores for each category, K Represents the total number of categories, softmax( xi ) indicates the first i The probability outputs for each category are calculated, with softmax_score representing the attention score and gate_score representing the gating score. Finally, a sigmoid gating mechanism is used to filter the composite channel scores, further suppressing noise bands while retaining effective signals, and finally obtaining the gating score.

[0037] By multiplying the attention score and the gating score to fuse the scores, the advantages of relative importance ranking are preserved while achieving absolute noise filtering. This operation ensures the relative proportion of key bands (such as the red-edge band) while further weakening the influence of noisy bands through gating mechanisms, resulting in more robust channel weights. These weights are then spatially weighted with V to preserve the spatial morphological characteristics of the target object and avoid spatial information loss due to channel filtering. Ultimately, this process generates the most valuable "spectral-spatial" joint features for classification. Where out represents the enhanced feature map after dual-channel attention modulation.

[0038] By concatenating the residual connections between the input and output feature maps: This operation integrates key features while preserving the original features. The `Concat` function concatenates multiple feature tensors along a specified dimension to form a new feature tensor with higher dimensions. This method avoids over-filtering by the attention mechanism and provides a rich spectral foundation for subsequent fusion. Subsequently, 1×1 convolutions are used for dimensionality reduction—reducing computational load while extracting deeper information through linear combination of cross-modal dual-channel features. This mechanism ensures the stability and smoothness of network learning, thereby improving the model's accuracy and generalization ability.

[0039] Finally, channel dimension normalization is applied to the fused features to stabilize their numerical distribution, effectively mitigating feature distribution shifts caused by band correlation in hyperspectral images, accelerating model convergence, and improving generalization ability. Next, ReLU, a simple and efficient nonlinear activation function, is applied to introduce nonlinearity by suppressing negative features while retaining positive features. Subsequently, nonlinear transformations are used to enhance the model's ability to express complex spectral features, further suppressing redundancy and focusing on key features, thereby achieving a lightweight module design. .

[0040] In this embodiment, the Global Band Channel Attention (GBCA) mechanism enhances feature discriminative power through compression and channel expansion, processing the initially extracted high-channel spectral spatial feature map while employing a gating mechanism to prevent attention distraction and irrelevant interference. After band filtering by the spectral band enhancer, GBCA achieves "feature enhancement" through channel attention scoring and gating mechanisms, resulting in discriminative spatial and spectral features. This mechanism complements the spectral filtering function of SBE, constructing a progressive optimization process from spectral selection to feature enhancement. This module strengthens the feature representation of key spectral bands for classification while effectively suppressing irrelevant frequency band interference, providing a robust spectral branching foundation for subsequent processing stages.

[0041] S4. Using learnable projection, the discriminative spatial features and discriminative spectral features are converted into fixed-length sequences to obtain spatial feature labels and spectral feature labels. Then, the spatial feature labels and spectral feature labels are fused using a gating enhancement token mechanism to obtain enhanced spatial labels and enhanced spectral labels.

[0042] The enhanced labeling method includes: In this embodiment, in the hyperspectral image classification task, the effective fusion of spectral and spatial features is crucial for improving classification accuracy. However, traditional fusion methods have two core problems: First, modal information imbalance—spectral features easily lose spatial scene context, while spatial features often ignore subtle spectral differences; second, rigid positional encoding—fixed positional encoding cannot adapt to the dynamic distribution characteristics of objects in hyperspectral data of different resolutions, making it difficult to capture the relationships between tokens in complex scenes. The core objectives of the GETA module proposed in this embodiment include: realizing bidirectional dynamic interaction between spectral and spatial tokens to enhance cross-modal feature complementarity; introducing dynamic relative positional encoding to adapt to the token distribution in multi-resolution data; and balancing the contribution of fused features and original features through gating mechanisms and learnable residual structures, thereby improving model stability.

[0043] To avoid attention bias caused by differences in feature distribution, we first normalize the input spectral feature tokens. Where b represents the batch size, L represents the number of tags, and d is the tag dimension; at the same time, the spatial feature tokens are also normalized. , here Q Origin TA K and V come from TB Subsequently, to optimize feature representation capabilities in order to generate... QFor example, we enhance nonlinearity through two-layer linear transformation and the GELU activation function, and introduce the Dropout mechanism to suppress overfitting. GELU is a smooth nonlinear activation function whose output is modulated by the probability distribution of the input value—when the input value increases, the activation output approaches the input itself; when the input value decreases, the activation output approaches 0. Dropout is a regularization technique in deep learning used to suppress overfitting. By randomly shutting down some neurons during training (setting their output to zero), it reduces excessive dependence between neurons. Its core principle is to enhance the model's generalization ability through randomness, as shown in the following formula: To address the challenges of cross-modal feature fusion, where attention mechanisms rely solely on content similarity, failing to capture the connections between semantically related but feature-dissimilar heterogeneous tokens, and feature fusion requirements exhibiting directional asymmetry and lack of order, asymmetric interaction modeling is needed. This paper introduces a spatial spectral adaptive guidance mechanism to initialize the learnable parameter matrix. Where Lmax is the preset maximum number of features. During training, the system truncates the submatrix according to the actual input feature token length L to avoid dimension mismatch issues.

[0044] The following formula represents the dimensionality adaptive pruning of the predefined relative position bias tensor `rel_pos_bias` to match the current input sequence length `L`. The pre-trained or predefined `rel_pos_bias` covers the maximum possible sequence length. By dynamically pruning to adapt to hyperspectral inputs of different sizes, the portion matching the current `L` is extracted from the predefined maximum-length bias tensor. A learnable bias term is provided for each pair of tokens to adjust their attention weights. This bias is independent of the input content and explicitly models the interaction preferences between token pairs. This achieves the shortcomings of pure content attention, enables efficient cross-modal adaptive semantic guidance, and replaces traditional relative position encoding, adapting to unordered token scenarios. Where rel_pos_biasadapt represents the relative positional relationship between features.

[0045] Now, we calculate the similarity between Q and K using matrix multiplication and divide by... (Where d represents the number of dimensions) to alleviate the score saturation phenomenon caused by dimensionality expansion, and then apply a dynamic positional bias to obtain the final attention score: Here, attn_scores represents the final attention score.

[0046] The final attention score is normalized using Softmax and Dropout to obtain the attention weights. These weights are then multiplied by the V matrix to generate intermediate features that fuse spatial information, as shown in the following formula: Here, attn_weights represents the attention weight matrix containing relative position information, i.e., how much attention should be paid, and attn_output represents the output after information fusion and attention weighting, i.e., the result obtained based on how much attention is paid.

[0047] To balance the contributions of cross-modal fusion features and original labeled features, this embodiment designs a dual adjustment mechanism to prevent either modality from dominating. After concatenating the original spectral labels and attention output, a series of operations, including linear transformation, are used to generate gated weights, thereby dynamically adjusting the fusion ratio. Here, "combined" means concatenating the original spectral feature input TA and the attention-enhanced output along the channel dimension to form a joint feature vector; "Linear" represents information compression and nonlinear transformation; "LayerNorm" means normalizing the feature vector of each feature token, i.e., the mean is 0 and the variance is 1, which improves training stability; and "gate" represents the gating weight of the newly generated joint feature vector, which determines the impact of "attn_output" on the original data.

[0048] Based on a gating weight mechanism, Gated Augmented Token Attention (GETA) achieves weighted deep fusion of fused features and original features, such as... Figure 5 As shown. This operation adaptively preserves key information from the original labels while incorporating cross-modal features: Subsequently, a learnable parameter α is introduced to perform weighted fusion between the original labels and the projected output features, thereby further improving the robustness of the model. By exchanging the sources of Q, K, and V, we generate enhanced spectral labels that fuse spatial feature information, and enhanced spatial labels that fuse spectral feature information, ultimately obtaining a cross-modal label representation with dual enhancement properties, resulting in the enhanced label. Specifically, the learnable parameter α... αThe model automatically adjusts its values ​​through backpropagation to optimize the fusion ratio between the original labels and projected features. This dynamic adjustment mechanism allows the model to flexibly allocate weights for different modal features based on the characteristics of the input data. Furthermore, by exchanging the sources of Q, K, and V, the model not only enhances its ability to represent spatial features in the spectral dimension but also enriches spectral feature information in the spatial dimension. This bidirectional enhancement strategy effectively improves the model's ability to understand and process complex cross-modal data. The calculation formula is as follows: in, TA `enhanced` indicates the enhanced spectral feature tokens with "introduced spatial context information," and `projected_output` represents the high-quality feature representation after nonlinear transformation, regularization, and dimensionality increase / decrease in the spectral branch, used for subsequent residual connections. TB "enhanced" indicates a spatial feature token enhanced by "introducing spectral context information". α’ Represents a learnable scalar parameter that controls the weight distribution between the original input and the newly generated features in the residual connection; projected_output ’ This represents a high-quality feature representation after nonlinear transformation, regularization, and dimensionality increase / decrease in the spatial branch.

[0049] S5. Generate image classification results based on enhanced spatial and spectral labels.

[0050] In this embodiment, dual-path classification is performed based on the enhanced spatial branch and spectral branch, and decision fusion is performed by combining learnable weight fusion and two-dimensional convolution fusion to generate image classification results.

[0051] Specifically, hyperspectral images contain rich spatial-spectral information, and their pixel classification must simultaneously capture spatial feature differences and subtle spectral distinctions between categories, making the classification task more complex than that of ordinary RGB images. To improve feature discrimination capabilities, this embodiment designs an efficient branch feature extraction structure and fusion strategy. For example... Figure 7 As shown, the feature extractor employs an innovative tokenization method to transform effective spatial and spectral features into a sequence of tags suitable for classification. These tags are cross-fused using the GETA mechanism to generate enhanced tags that simultaneously contain both spatial and spectral information. The two types of enhanced tags are flattened into one-dimensional feature vectors, and after layer normalization, their respective linear classifiers generate preliminary classification results for spectral and spatial branches.

[0052] To improve the fusion effect, this network adopts a dual-path decision layer strategy, combining learnable weight fusion and two-dimensional convolutional fusion. First, the preliminary classification results from the two branches are fused using learnable weights; simultaneously, the concatenated preliminary results are subjected to two-dimensional convolution to achieve deep fusion. Finally, the two fused outputs are averaged to generate a one-dimensional classification result. This method has the following advantages: learnable weight fusion can dynamically adjust branch weights and adaptively allocate classification dominance; two-dimensional convolutional fusion can uncover the correlation between results, capturing potential information through cross-branch mapping and integrating it into the global decision, providing richer judgment criteria; the combined effect of the two methods can alleviate overfitting, making the final result possess both linear weighted stability and the feature interaction capability of convolutional fusion. Furthermore, averaging improves the judgment accuracy in difficult regions, and dynamic adaptation and global correction mechanisms enhance decision robustness. The dual-path decision layer strategy overcomes the limitations of single fusion methods in feature representation, ensuring that the final decision features have stable class discriminative power, richer cross-branch correlation, and stronger generalization ability.

[0053] Taking spectral branching as an example, we analyze the process of spectral feature enhancement. In spectral branching, the core function of labeling is to compress high-dimensional spectral-spatial features into discriminative labeled features, providing an efficient feature representation for subsequent classification. Similarly, for spatial branching features, The input is a feature tensor, where B represents the batch size, C represents the number of spectral bands, and H and W are the image height and width, respectively. The input Res is reshaped into... The attention weights between the spectral features and the tokens are then calculated. This process essentially measures the association strength between each spatial pixel (N-dimensional) and each feature token (L-dimensional). This operation performs batch matrix multiplication, multiplying the Res of each sample by the transpose of token_wAx: in, Ax This represents the attention weights, and einsum represents the einsum function. The operation between tensors is essentially to calculate the similarity between each token and each spatial location.

[0054] The spatial pixels of hyperspectral data contain rich topographic details. This operation reveals the correlation between pixels and markers, laying the foundation for subsequent aggregation of key spatial information. This represents the learned weight matrix, where L represents the number of labels. The `einsum` function performs specific matrix multiplication operations to generate the result. Subsequently, the spatial pixel dimension is normalized using softmax to ensure that the sum of the weights for each label is 1. This mechanism highlights the spatial pixels that contribute most significantly to each label while suppressing irrelevant spatial noise, focusing on key spatial regions strongly correlated with spectral features, thereby improving feature robustness. Where dim represents taking Ax The first dimension is the spatial dimension. This step generates normalized attention weights. By calculating the dot product of the input feature Res and another learned weight matrix token_wVx, the spectral features are projected from the channel dimension to the label dimension, generating value features that can be aggregated from the feature tokens. in, VVx Indicates value characteristics, The operation between tensors essentially involves projecting the c-dimensional features of each batch and each spatial location onto a d-dimensional feature space. , CT It is the feature dimension of the token, which is obtained By compressing the feature dimension through linear projection while preserving the discriminative information of spectral features (such as key band combinations), high-quality input is provided for token aggregation. Attention weights are used ( Ax ) logarithmic features ( VVx Perform a weighted summation to generate the final token characteristics: in, This represents the weighted feature representation. This means using attention weights to perform a weighted average of the value features, generating L output tokens. Compressing high-dimensional spatial-spectral features into a compact token sequence preserves key information while significantly reducing computational cost. The spatial branching results, which also undergo the same operation, are then used... Ty By performing cross-fusion, an enhanced feature representation combining spatial information is obtained. Tx The weighted features are flattened and classified through a linear layer for the spectral branch. This operation achieves the core of feature purification and compression for the spectral branch. A tokenization mechanism balances information preservation and computational efficiency of hyperspectral data, providing the model with a robust spectral-spatial joint feature representation, a key step in achieving high-precision classification. Finally, the classification results from the spectral and spatial branches are fused using a learnable weighted method, supplemented by 2D convolutional fusion. The resulting output is a more robust and accurate hyperspectral image classification result. This fully realizes the optimization of the entire process—feature purification, cross-branch fusion, and multi-path decision-making—fully leveraging the synergistic advantages of the tokenization mechanism and dual-path fusion, providing an efficient and reliable solution for hyperspectral image classification tasks.

[0055] Example 2 In this embodiment, a hyperspectral image classification system based on a spectral-spatial dual-branch token feature fusion network includes: an image preprocessing module, a feature fusion module, a feature extraction module, a feature fusion module, and a result generation module.

[0056] The image preprocessing module is used to acquire the original hyperspectral image, preprocess the original hyperspectral image, and perform joint spectral-spatial feature extraction on the preprocessed image to obtain three-dimensional output features.

[0057] The workflow of the image preprocessing module includes: using principal component analysis to reduce the dimensionality of the original hyperspectral image to obtain dimensionality-reduced image data; using 3D convolution to perform preliminary spatial and spectral feature extraction on the dimensionality-reduced image data to obtain initial spectral features; and using a spectral band enhancer to perform pooling and stitching processing on the initial spectral features to obtain 3D output features.

[0058] The feature fusion module is used to perform global average pooling on the 3D output features along the spectral dimension to obtain 2D features containing global contextual information. At the same time, the 3D output features are flattened in the channel and spectral dimensions to preserve local spatial spectral details. Then, the 2D features and local spatial spectral details are concatenated in the channel dimension to construct complete spatial and spectral features that integrate local details and global semantics.

[0059] The feature extraction module enhances key spatial features in spatial and spectral features by utilizing global grouped coordinate attention, and optimizes spectral features in spatial and spectral features by utilizing gated bottleneck channel attention, thereby obtaining discriminative spatial features and discriminative spectral features.

[0060] In the feature extraction module, the process of obtaining discriminative spatial features includes: dividing the spatial and spectral features into several groups, and performing global average pooling and global max pooling on the feature map of each group to obtain pooled feature maps; performing convolution operations on the pooled feature maps to extract local features to obtain output feature maps; calculating the height direction attention weight and the width direction attention weight based on the output feature maps, and obtaining discriminative spatial features based on the spatial and spectral features, the height direction attention weight, and the width direction attention weight.

[0061] In the feature extraction module, the process of obtaining discriminative spectral features includes: processing spatial and spectral features using three parallel bottleneck structures to obtain channel features; calculating attention scores and gating scores based on channel features, and obtaining spectral-spatial joint features based on attention scores and gating scores; fusing spatial and spectral features and spectral-spatial joint features, and normalizing the channel dimensions to obtain discriminative spectral features.

[0062] The feature fusion module uses learnable projection to convert discriminative spatial features and discriminative spectral features into fixed-length sequences, obtaining spatial feature labels and spectral feature labels. Then, it uses a gating enhancement token mechanism to fuse the spatial feature labels and spectral feature labels to obtain enhanced spatial labels and enhanced spectral labels.

[0063] The results generation module generates image classification results based on enhanced spatial and spectral labels.

[0064] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A hyperspectral image classification method based on a spectral-spatial dual-branch token feature fusion network, characterized in that, Includes the following steps: The original hyperspectral image is acquired, preprocessed, and preliminary extraction of joint spectral-spatial features is performed on the preprocessed image to obtain three-dimensional output features. The three-dimensional output features are globally averaged along the spectral dimension to obtain two-dimensional features containing global contextual information. At the same time, the three-dimensional output features are flattened in the channel and spectral dimensions to preserve local spatial spectral details. The two-dimensional features and the local spatial spectral details are then concatenated in the channel dimension to construct complete spatial and spectral features that integrate local details and global semantics. By using global grouped coordinate attention to enhance key spatial features in the spatial and spectral features, and by using gated bottleneck channel attention to optimize spectral features in the spatial and spectral features, discriminative spatial features and discriminative spectral features are obtained. The discriminative spatial features and discriminative spectral features are converted into fixed-length sequences using learnable projection to obtain spatial feature tags and spectral feature tags. The spatial feature tags and spectral feature tags are then fused using a gated enhancement token mechanism to obtain enhanced spatial tags and enhanced spectral tags. Image classification results are generated based on the enhanced spatial label and the enhanced spectral label.

2. The hyperspectral image classification method based on a spectral-spatial dual-branch token feature fusion network according to claim 1, characterized in that, Methods for obtaining 3D output features include: Principal component analysis was used to reduce the dimensionality of the original hyperspectral image to obtain the dimensionality-reduced image data. The dimensionality-reduced image data is preliminarily extracted using three-dimensional convolution to obtain initial spectral features; The initial spectral features are pooled and stitched using a spectral band enhancer to obtain the three-dimensional output features.

3. The hyperspectral image classification method based on a spectral-spatial dual-branch token feature fusion network according to claim 1, characterized in that, The method for obtaining the discriminative spatial features includes: The spatial and spectral features are divided into several groups, and the feature maps of each group are subjected to global average pooling in the height dimension and global max pooling in the width dimension to obtain pooled feature maps. Convolution operations are performed on the pooled feature maps respectively to extract local features and obtain the output feature map; The height-direction attention weight and width-direction attention weight are calculated based on the output feature map, and the discriminative spatial features are obtained based on the spatial and spectral features, the height-direction attention weight, and the width-direction attention weight.

4. The hyperspectral image classification method based on a spectral-spatial dual-branch token feature fusion network according to claim 1, characterized in that, The method for obtaining the discriminative spectral features includes: The spatial and spectral features are processed using three parallel bottleneck structures to obtain channel features; The attention score and gating score are calculated based on the channel features, and the spectral-spatial joint features are obtained based on the attention score and the gating score. The spatial and spectral features and the spectral-spatial joint features are fused and the channel dimensions are normalized to obtain the discriminative spectral features.

5. A hyperspectral image classification system based on a spectral-spatial dual-branch token feature fusion network, wherein the system applies the method described in any one of claims 1-4, characterized in that, include: Image preprocessing module, feature fusion module, feature extraction module, feature fusion module, and result generation module; The image preprocessing module is used to acquire the original hyperspectral image, preprocess the original hyperspectral image, and perform joint spectral-spatial feature extraction on the preprocessed image to obtain three-dimensional output features; The feature fusion module is used to perform global average pooling on the three-dimensional output features along the spectral dimension to obtain two-dimensional features containing global context information, and simultaneously flatten the three-dimensional output features in the channel and spectral dimensions to preserve local spatial spectral details. Then, the two-dimensional features and the local spatial spectral details are concatenated in the channel dimension to construct complete spatial and spectral features that fuse local details and global semantics. The feature extraction module enhances key spatial features in the spatial and spectral features by using global grouped coordinate attention, and optimizes spectral features in the spatial and spectral features by using gated bottleneck channel attention, thereby obtaining discriminative spatial features and discriminative spectral features. The feature fusion module uses learnable projection to convert the discriminative spatial features and the discriminative spectral features into fixed-length sequences, respectively, to obtain spatial feature labels and spectral feature labels. Then, it uses a gated enhancement token mechanism to fuse the spatial feature labels and the spectral feature labels to obtain enhanced spatial labels and enhanced spectral labels. The result generation module generates image classification results based on the enhanced spatial label and the enhanced spectral label.

6. The hyperspectral image classification system based on a spectral-spatial dual-branch token feature fusion network according to claim 5, characterized in that, The workflow of the image preprocessing module includes: Principal component analysis was used to reduce the dimensionality of the original hyperspectral image to obtain the dimensionality-reduced image data. The dimensionality-reduced image data is preliminarily extracted using three-dimensional convolution to obtain initial spectral features; The initial spectral features are pooled and stitched using a spectral band enhancer to obtain the three-dimensional output features.

7. The hyperspectral image classification system based on a spectral-spatial dual-branch token feature fusion network according to claim 5, characterized in that, The process of obtaining the discriminative spatial features in the feature extraction module includes: The spatial and spectral features are divided into several groups, and the feature maps of each group are subjected to global average pooling in the height dimension and global max pooling in the width dimension to obtain pooled feature maps. Convolution operations are performed on the pooled feature maps respectively to extract local features and obtain the output feature map; The height-direction attention weight and width-direction attention weight are calculated based on the output feature map, and the discriminative spatial features are obtained based on the spatial and spectral features, the height-direction attention weight, and the width-direction attention weight.

8. The hyperspectral image classification system based on a spectral-spatial dual-branch token feature fusion network according to claim 5, characterized in that, The process of obtaining the discriminative spectral features in the feature extraction module includes: The spatial and spectral features are processed using three parallel bottleneck structures to obtain channel features; The attention score and gating score are calculated based on the channel features, and the spectral-spatial joint features are obtained based on the attention score and the gating score. The spatial and spectral features and the spectral-spatial joint features are fused and the channel dimensions are normalized to obtain the discriminative spectral features.

Images