A hyperspectral image classification method and device based on frequency-semantic synergistic modulation

Abstract

The application discloses a hyperspectral image classification method and device based on frequency-semantics collaborative modulation. The method comprises inputting a hyperspectral image and performing block processing to obtain multiple groups of hyperspectral image subblocks and corresponding input texts. Spatial-spectral interaction blocks are used to extract features of the hyperspectral image subblocks to obtain spatial-spectral fusion features. An RGB-guided frequency-spatial embedding block is used to perform frequency-spatial collaborative modulation on the spatial-spectral fusion features based on an RGB image to obtain multi-directional frequency-spatial enhanced features. An adaptive multi-scan text guided Mamba module is used to perform semantic guidance and multi-directional state space modeling on the multi-directional frequency-spatial enhanced features based on the input texts to obtain final classification features. The hyperspectral image is classified based on the final classification features, and a total loss function is constructed through semantic alignment loss and classification loss to complete model training and classification output. The method has high calculation efficiency, strong robustness and good semantic consistency.

Description

Technical Field

[0001] This invention belongs to the field of intelligent image processing and computer vision technology, specifically relating to a hyperspectral image classification method and apparatus based on frequency-semantic co-modulation. Background Technology

[0002] Hyperspectral images (HSI) possess high-dimensional spectral information and rich spatial structure, making them invaluable in geological exploration, precision agriculture, and environmental monitoring. Existing hyperspectral image classification methods mainly include: methods based on traditional machine learning (such as SVM), but these rely on manual feature design and struggle to characterize complex spatial-spectral relationships; methods based on convolutional neural networks, but these are limited by local receptive fields and struggle to model long-distance inter-spectral dependencies; methods based on Transformers, while possessing global modeling capabilities, have high computational complexity and are unsuitable for high-dimensional hyperspectral data; and methods based on state-space models (such as Mamba), while offering advantages in linear complexity, typically employ unidirectional scanning, limiting state propagation paths and making it difficult to fully model multi-path dependencies in two-dimensional space.

[0003] Furthermore, existing frequency domain methods typically model frequency features uniformly, lacking a structured grouping mechanism, which makes it difficult to utilize high-frequency and low-frequency information in a coordinated manner; existing multimodal methods mostly adopt feature splicing or attention fusion methods, lacking a structured modulation mechanism for the feature propagation process, and cannot effectively suppress spectral confusion. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, the present invention aims to provide a hyperspectral image classification method and apparatus based on frequency-semantic co-modulation. This method solves the problems of locality, high complexity, or unidirectional propagation limitations in spatial-spectral dependency modeling, and the lack of structured coordination and modulation mechanisms in frequency domain utilization and multimodal feature fusion. It achieves efficient decoupling of spatial-spectral features, long-distance inter-spectral dependency modeling, spectral confusion suppression, and improved generalization ability for small samples.

[0005] This invention is achieved through the following technical solution: A hyperspectral image classification method based on frequency-semantic co-modulation includes the following steps: S1. Input a hyperspectral image, perform block processing on the hyperspectral image to obtain multiple sets of hyperspectral image sub-blocks, and construct the corresponding input text; S2. Feature extraction is performed on the hyperspectral image sub-blocks through spatial-spectral interaction blocks. The spatial-spectral interaction blocks construct a dual-branch structure through three-dimensional convolution and two-dimensional convolution to jointly model spatial information and spectral information, thereby obtaining spatial-spectral fusion features. S3. Using RGB-guided frequency-spatial embedding blocks, frequency-spatial co-modulation is performed on the spatial-spectral fusion features based on the RGB image to obtain multi-directional frequency-spatial enhancement features; S4. The Mamba module is guided by adaptive multi-scan text to perform semantic guidance and multi-directional state space modeling on the multi-directional frequency-space enhancement features based on the input text, so as to obtain the final classification features. S5. Perform hyperspectral image classification based on the final classification features, and construct a total loss function through semantic alignment loss and classification loss to complete model training and classification output.

[0006] Furthermore, in step S2, the feature extraction process of the spatial-spectral interaction block includes: For each input hyperspectral image sub-block, parallel feature extraction is performed using both 3D and 2D convolutional branches: 3D Convolution Branch: Extract spatial-spectral joint features by sequentially passing through 3×3×3 3D convolution, batch normalization, GELU activation function, 3×3×N 3D convolution, batch normalization, and GELU activation function; 2D convolution branch: Spatial dimension features are extracted sequentially through 3×3 2D convolution, batch normalization, and GELU activation function; The output features of the 3D convolutional branch and the 2D convolutional branch are fused by channel reweighting to obtain spatial-spectral fusion features.

[0007] Furthermore, in step S3, the modulation process of the RGB-guided frequency-space embedding block includes: S3.1 Perform a two-dimensional fast Fourier transform on the spatial-spectral fusion features to obtain frequency domain complex-valued features; decompose the frequency domain complex-valued features into real features and imaginary features; divide the spectral channels into several groups, and perform group-level learnable transformation and activation processing on the real features and imaginary features respectively; perform residual fusion on the processed real features and imaginary features and reassemble them into new frequency domain complex-valued features; perform a two-dimensional inverse fast Fourier transform on the new frequency domain complex-valued features to obtain frequency enhancement features; S3.2 For the corresponding RGB image, generate RGB guided modulation features by sequentially passing two layers of 3×3 convolution, batch normalization, and ReLU activation function; S3.3 Multiply the frequency enhancement features and RGB guided modulation features element by element, and then perform deformable convolution enhancement and root mean square normalization in sequence to obtain multi-directional frequency-space enhancement features.

[0008] Furthermore, in step S4, the process of self-adapting multi-scan text to bootstrap the Mamba module includes: S4.1 Perform 3D conditional position encoding on the input multi-directional frequency-spatial enhancement features and inject spatial spectral position information; normalize and linearly project the encoded features to obtain state-space modeling branch features and gated branch features, respectively. S4.2. Convolution and activation are performed on the state space modeling branch features to enhance the local context. The text encoder is used to extract the category text embedding. Semantic gating vectors are generated through linear projection and activation. The state space modeling branch features are modulated element by element. Multi-directional selective state space scanning is performed on the modulated features to obtain multiple directional scanning features. S4.3. The gated branch features are linearly projected and activated to generate adaptive fusion weights. The multiple directional scanning features are weighted and fused, and then linearly projected to output the final classification features.

[0009] Furthermore, in step S5, the formula for constructing the total loss function is: in, The classification loss is used to supervise the classification accuracy of hyperspectral images; λ is the semantic alignment loss, used to constrain the spatial-semantic consistency between text semantic features and visual features; λ is the balancing weight coefficient.

[0010] Furthermore, the input text is a category semantic description text corresponding to the hyperspectral image scene, used to provide semantic supervision guidance for visual feature extraction.

[0011] Furthermore, the AdamW optimizer is used for model training, with an initial learning rate of 1e-5 to 1e-3, a weight decay of 1e-3 to 1e-1, chord annealing learning rate scheduling, a warm-up of 3 to 8 epochs, a total training epoch of 40 to 60, and a batch size of 2 to 8.

[0012] Furthermore, the multi-directional selective state space scanning includes at least two of the row direction, column direction, and diagonal direction.

[0013] A hyperspectral image classification device based on frequency-semantic co-modulation includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the image classification method.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: Through a cascaded design of the spatial-spectral interaction block SSIB, the RGB-guided frequency-spatial embedding module RG-FSEM, and the adaptive multi-scan text-guided Mamba module TMSM, deep fusion of spatial-spectral features, frequency features, and semantic features is achieved. This has been demonstrated at Indian Pines, Pavia University, and Salinas. On three major public benchmark datasets, the overall classification accuracy (OA) reaches over 99.89%, and the Kappa coefficient reaches over 98.45%, significantly outperforming SVM, CNN, Transformer, and existing Mamba methods. Specifically, the linear time series modeling based on Mamba / SSM, combined with multi-directional selective scanning, avoids the secondary computational complexity of Transformer. Furthermore, frequency grouping and channel reweighting suppress redundant computation, improving computational efficiency and scalability while maintaining accuracy, resulting in high computational efficiency. Multi-directional spectral scanning and frequency-aware modeling effectively solve the problems of spectral confusion and spatial structure ambiguity, producing classification maps with clearer class boundaries and more uniform regions. It maintains excellent performance and strong robustness even in scenarios with high spectral similarity, class imbalance, and small sample sizes. The introduction of a class-level semantic alignment mechanism binds visual features to textual semantic prototypes, improving feature discriminativeness and resolving the inconsistency between visual and semantic features in existing methods, resulting in good semantic consistency. This method is applicable to hyperspectral images in various scenarios (agriculture, urban areas, geology), requiring no extensive parameter tuning for specific datasets. It can be directly transferred to practical applications such as geological surveying, precision agriculture, and environmental monitoring, demonstrating strong versatility and promising engineering prospects. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall architecture of the LM³-HSI framework of the present invention; Figure 2 This is a detailed schematic diagram of the module composition and process flow of the LM³-HSI framework of the present invention; Figure 3 This is a schematic diagram of the selective state space scanning mechanism in the TMSM module of the present invention; Figure 4 This is a schematic diagram comparing the performance of different scanning strategies in the TMSM module of the present invention; Figure 5 This is a schematic diagram comparing the classification graphs of the present invention and existing methods on the IndianPines dataset; Figure 6 This is a schematic diagram comparing the classification graphs of the present invention and existing methods on the Pavia University dataset; Figure 7 This is a schematic diagram comparing the classification graphs of the present invention and existing methods on the Salinas dataset; Figure 8 is a schematic diagram of the t-SNE features under different ablation settings of the present invention. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0017] like Figure 1 The diagram shown is the overall architecture of the LM³-HSI unified multi-scale semantic perception hyperspectral learning framework. This diagram intuitively presents the core design logic of the LM³-HSI framework of this invention, clearly corresponds to the three core technical challenges, targeted solutions and technical benefits in the field of hyperspectral image classification, and demonstrates the complete workflow of the model from input to output. The left side of the figure clearly identifies three major pain points in existing technologies: multi-scale inter-spectral dependencies and spectral confusion, irregular and multi-scale spatial structures, and lack of cross-scale semantic consistency. The middle column addresses these pain points by proposing the design of three core modules of this invention: the Adaptive Multi-Scan Text-Guided Mamba Module (TMSM), the RGB-Guided Frequency-Spatial Embedding Module (RG-FSEM), and the Class-Level Semantic Alignment Mechanism. The core design ideas of each module are clarified as multi-scale inter-spectral modeling + semantic guidance, frequency awareness + RGB-guided deformable sampling, and semantic anchor feature learning based on frozen text encoders. The right column marks the technical benefits of each module in solving the corresponding pain points, namely robust long-range inter-spectral modeling, adaptive spatial representation, and semantically consistent features. The bottom of the figure shows the end-to-end execution flow of the framework, which takes hyperspectral image patches, RGB image patches, and text prompts as joint inputs, and processes them sequentially through the Spatial-Spectral Interaction Block (SSIB), RG-FSEM, and TMSM modules. Finally, it outputs predicted land cover labels through Global Average Pooling (GAP) + Fully Connected Layer (FC).

[0018] Based on this overall architecture, this invention achieves four core innovative designs: First, it designs a dual-branch SSIB architecture to achieve efficient coupling of local spatial-spectral features, suppress spectral redundancy, and preserve spatial continuity. Second, it proposes an RG-FSEM module that integrates frequency domain grouping modeling and RGB-guided spatial modulation, balancing local texture (high frequency) and global structure (low frequency). Third, it proposes a TMSM module that employs selective SSM with multi-directional spectral scanning to achieve long-distance inter-spectral dependency modeling with linear time complexity, combined with text semantic gating to suppress spectral confusion. Fourth, it introduces a class-level semantic alignment mechanism for freezing multimodal text encoders to improve the semantic consistency of visual features and enhance generalization ability under small sample sizes.

[0019] like Figure 2The diagram shows the detailed module composition and flow of the LM³-HSI framework of this invention. Following the feature processing flow, the diagram sequentially displays the complete hierarchical structure of the input module → Spatial-Spectral Interaction Block (SSIB) → RGB Guided Frequency-Spatial Embedding Module (RG-FSEM) → Adaptive Multi-Scan Text Guided Mamba Module (TMSM). The core computational units of each module are clearly labeled: 3D convolution and 2D convolution in SSIB; FFT / IFFT, deformable convolution, and RMSnorm in RG-FSEM; and 3D-CPE, selective SSM scanning, and linear projection in TMSM. The diagram also clarifies the cross-module fusion relationship of multimodal semantic guidance, demonstrating how the class-level semantic prototypes extracted by the frozen large-scale multimodal text encoder provide text gating modulation for the TMSM module, and how the semantic cues of the RGB image provide spatial modulation basis for the RG-FSEM module. This intuitively reflects the core design concept of "multimodal semantic guidance" of this invention. The diagram also labels the joint optimization logic of the loss function, clearly defining the classification loss (…). ) and semantic alignment loss ( The method employs a weighted fusion approach, distinguishing between the training and inference phases of the model. The text encoder, which is responsible for labeling semantic alignment, only participates in the training phase and is removed in the inference phase. This approach ensures the consistency of feature semantics during training and reduces the computational overhead in the inference phase, thereby improving the engineering practicality of the method.

[0020] like Figure 3 The diagram illustrates the structure of the selective state-space scanning mechanism in the TMSM module of this invention. It showcases the four scanning methods of the selective state-space scanning mechanism (SSM) (Mamba) within the TMSM module and the design logic of the shared SSM, clearly demonstrating how this mechanism overcomes the limitations of traditional single-scanning methods to fully explore multi-scale, long-distance spectral dependencies. The diagram shows four differentiated spectral scanning sequences: row-wise zigzag scanning, column-wise zigzag scanning, diagonal scanning, and anti-diagonal scanning. The feature traversal paths for each scanning method are marked, reflecting the core design of multi-directional scanning in this invention. The key design of "shared selective SSM" is also highlighted. The four scanning methods are implemented based on the same SSM model, ensuring comprehensive exploration of multi-scale spectral features while avoiding the additional computational overhead of parallel multi-model operation, achieving efficient modeling. By covering different feature association angles across spectral dimensions through multi-directional scanning, long-distance dependencies across bands can be fully captured, effectively solving the technical problems of insufficient information mining and spectral confusion inherent in traditional unidirectional / bidirectional scanning. This provides visual technical support for the design of the TMSM module.

[0021] The hyperspectral image classification method of this invention is implemented based on the PyTorch framework, with an Intel Xeon Gold CPU and an NVIDIA A100 GPU. The specific steps are as follows: S1. Hyperspectral Image Preprocessing Input raw hyperspectral image ,in For space dimensions, The number of spectral bands; an overlapping block strategy is adopted, and the block size is... Select 16-64 pixels (preferably 32 pixels) with an overlap rate of 20%-50% (preferably 30%) to obtain a hyperspectral block set. , Normalize the hyperspectral block to eliminate dimensional differences between bands.

[0022] S2. Feature extraction is performed on the hyperspectral image sub-blocks through spatial-spectral interaction blocks. The spatial-spectral interaction blocks construct a dual-branch structure through three-dimensional convolution and two-dimensional convolution to jointly model spatial information and spectral information, thereby obtaining spatial-spectral fusion features. For each input hyperspectral image sub-block, parallel feature extraction is performed using both 3D and 2D convolutional branches: 3D spatial-spectral joint feature branch: 3×3×3 3D convolution + BN + GELU activation is used sequentially, followed by 3×3×N 3D convolution (N is 8-32, preferably 16) + BN + GELU activation to achieve cross-band integration and local spatial-spectral feature mining. 2D Spatial Feature Branch: Employs 3×3 2D convolution + BN + GELU activation to capture local spatial texture and edge information; Channel Reweighted Fusion: Adds / concatenates the dual-branch features element-wise, then reweights the channels using the SE attention mechanism to output SSIB features. .

[0023] S3. Using an RGB-guided frequency-spatial embedding block, frequency-spatial co-modulation is performed on the spatial-spectral fusion features based on the RGB image to obtain multi-directional frequency-spatial enhancement features. Frequency group grouping modeling: for Do The transformation yields the complex-valued feature T, which is decomposed into its real part. and the virtual part Divide the spectral channels into G groups (G ranges from 4 to 16, preferably 8), and perform group-level learnable transformations on the real and imaginary components respectively (weight matrix). bias GELU activation, residual connections followed by recombination Frequency enhancement feature x_freq is obtained through inverse FFT; RGB-guided spatial modulation: Global average pooling (GAP) + 1D convolution (kernel size 1-3) + Sigmoid activation are performed on the corresponding RGB image to generate RGB guided modulation features M_RGB; Will and Element-wise multiplication, followed by deformable convolution (3×3) + RMSnorm normalization, outputs RG-FSEM features. .

[0024] S4. The Mamba module is guided by adaptive multi-scan text to perform semantic guidance and multi-directional state space modeling on the multi-directional frequency-space enhancement features based on the input text, so as to obtain the final classification features. 3D position encoding: for Perform 3D conditional position coding (CPE3D), inject spatial spectral position information, and obtain Feature projection and gating: for Do Normalization, obtained by linear projection (State-space modeling branch) and (Gated branch); Local context is enhanced using 3×3 convolution with SiLU activation; class-level text embeddings are extracted using a frozen CLIP text encoder. After linear projection + Activation generates a semantically gated vector g_t, which is then modulated element-wise on u to obtain û; multi-directional SSM scan: for Perform selective SSM scans in 4-8 directions (preferably 4: row, column, diagonal, and anti-diagonal) to obtain... Scan features ; Adaptive fusion: linear projection onto v + Activate the generation of fusion weights ,right Weighted fusion is obtained Output via linear projection feature .

[0025] S5. Perform hyperspectral image classification based on the final classification features, and construct a total loss function through semantic alignment loss and classification loss to complete model training and classification output; Visual feature aggregation: Visual features obtained by performing GAP Obtained by linear projection (Consistent with the text embedding dimension); Semantic prototype construction: Input textual descriptions of land cover categories (such as forests, farmland, and buildings) into a frozen CLIP / BERT text encoder to generate class-level semantic prototypes. ; Joint optimization of loss functions: semantic alignment loss : Using comparative loss, temperature hyperparameter The value is set to 0.05-0.2 (preferably 0.1) to maximize the similarity of the target class and suppress the similarity of non-target classes; classification loss. Using cross-entropy loss, for Do Obtain class probability; total loss λ is 0.5-2 (preferably 1); Optimizer settings: The AdamW optimizer is used, with an initial learning rate of 1e-5-1e-3 (preferably 1e-4), a weight decay of 1e-3-1e-1 (preferably 1e-2), cosine annealing for learning rate scheduling, a warm-up period of 3-8 epochs (preferably 5), a total training epoch of 40-60 (preferably 50), and a batch size of 2-8 (preferably 2 / 4). Inference and classification map generation: After training, the test hyperspectral block is input into the model to obtain the classification result, and the classification map of the entire hyperspectral image is obtained by fusion of overlapping blocks.

[0026] Figure 4 This diagram illustrates the performance comparison of different scanning strategies in the TMSM module of this invention. The graph, presented as a bar chart, quantitatively displays the performance differences of four scanning strategies—row, column, diagonal, and anti-diagonal—on three core indicators: overall accuracy (OA), average accuracy (AA), and Kappa coefficient. The results clearly show that the classification indicators for diagonal and anti-diagonal scanning are significantly higher than those for row and column scanning. This diagram clarifies the experimental basis for selecting multi-directional scanning (including diagonal / anti-diagonal) in this invention and explains the technical reasons for the optimal scanning strategy: diagonal / anti-diagonal scanning can simultaneously cover the lateral and vertical correlations of the spectral dimension, more fully exploring long-distance dependencies across bands, while row / column scanning can only capture single-dimensional feature correlations, resulting in insufficient information mining.

[0027] The feasibility of spatial-spectral feature coupling in this invention: SSIB's 3D convolution can simultaneously model spatial and spectral dimensions, 2D convolution supplements pure spatial texture, dual-branch fusion takes into account both spatial-spectral joint features and spatial details, and the channel reweighting mechanism adaptively suppresses redundant features, which conforms to the feature distribution law of "spatial-spectral integration" in hyperspectral images. The feasibility of frequency-aware modeling in this invention: Based on the Fourier analysis principle, the high-frequency components of hyperspectral images correspond to local textures / boundaries, while the low-frequency components correspond to global structures. Frequency grouping modeling of RG-FSEM enables targeted learning of different frequency features. RGB-guided deformable convolution adaptively adjusts the receptive field to match complex spatial structures, thus solving the problem of the disconnect between frequency features and spatial structures. The feasibility of multi-scan Mamba modeling in this invention: The state-space model of Mamba / SSM can realize long sequence modeling with linear time complexity. Multi-directional scanning breaks through the information bottleneck of single scanning and fully explores long-distance dependence across bands. Text semantic gating suppresses spectral confusion and improves inter-class discriminability from the semantic level by learning high-dimensional semantic prior constraint features. Feasibility of semantic alignment in this invention: The frozen large-scale multimodal text encoder has learned rich language-visual semantic associations. Using its class-level semantic prototypes as anchors, hyperspectral visual features are aligned through contrastive loss, realizing the feature fusion of "semantics-visual" and solving the problem of insufficient feature discriminability under small sample conditions.

[0028] The modules of this invention form a complementary and synergistic whole: SSIB is the foundation for initial coupling of spatial-spectral features; RG-FSEM is the enhancement for in-depth mining of frequency-spatial features; TMSM is the core for semantic-guided modeling of long-distance spectral dependencies; and semantic alignment is the constraint to improve the semantic consistency of features, ultimately achieving accurate and robust classification of hyperspectral images.

[0029] Figure 5 This diagram illustrates a comparison of classification maps between the method of this invention and existing methods on the Indian Pines dataset. This dataset is a hyperspectral dataset of agricultural scenes, characterized by high spectral similarity and class imbalance, making it a classic and challenging dataset for hyperspectral classification. The diagram compares the classification results of SVM, 2D-CNN, 3D-CNN, existing Mamba-type methods, Transformer-type methods, and the LM³-HSI method of this invention, using the hyperspectral false-color original image and the ground truth image as references. The differences in classification performance between the methods are clearly demonstrated. The classification map generated by the method of this invention has clearer boundaries between land cover categories and more uniform region division. In areas with high spectral similarity, such as crop categories and small sample categories, there are almost no misclassified or missed pixels. In contrast, the comparative methods generally suffer from salt noise, blurred category boundaries, and misclassification in local areas. This diagram visually verifies the superiority of the method of this invention in hyperspectral image classification of agricultural scenes.

[0030] Figure 6This diagram illustrates a comparison of the classification results of the proposed method and existing methods on the Pavia University dataset. This dataset is a hyperspectral dataset of urban scenes, characterized by high spatial resolution, complex terrain structures, and numerous irregular boundaries, demanding a high level of spatial feature modeling capability from the model. The diagram uses the false-color original image and the ground truth label image as references to compare the classification results of various mainstream methods. In the boundary areas of complex terrain features such as buildings, roads, vegetation, and bare soil, the proposed method's classification results closely match the ground truth labels, with no significant class confusion. In contrast, the comparative methods are prone to misclassification in complex spatial structures. This diagram verifies that the proposed method, through the spatial adaptive design of the RG-FSEM module, can effectively adapt to the complex spatial structures of urban scenes, improving the accuracy and robustness of classification.

[0031] Figure 7 This diagram illustrates a comparison of the classification results of the method of this invention with existing methods on the Salinas dataset. This dataset is a large-scale hyperspectral dataset of agricultural scenes, with well-organized ground feature blocks and sufficient labeled samples, which can verify the feature discrimination and generalization capabilities of the model in large-sample scenarios. The comparison results show that the method of this invention achieves accurate classification results and good regional uniformity for crops at different growth stages and different types of soil / vegetation areas, fully capturing the spectral and spatial feature differences of ground features. This diagram verifies the effectiveness of the method of this invention in large-scale hyperspectral image classification and also demonstrates the versatility of the method, which can be adapted to hyperspectral image classification tasks of different types and scenarios.

[0032] This invention was experimentally validated on three major internationally available hyperspectral datasets: Indian Pines (IP), Pavia University (PU), and Salinas (SA). It was compared with over ten mainstream methods, including SVM, RBF-SVM, 2D-CNN, 3D-CNN, HSI-BERT, HyperMamba, and HSI-MFormer. The core quantification metrics are overall accuracy (OA), average accuracy (AA), and Kappa coefficient. Key experimental data are as follows: IndianPines dataset: The present invention achieves OA=99.96%, AA=99.38%, and Kappa=98.45%, which is 1.09 percentage points higher than the suboptimal method HyperMamba (OA=98.87%). Table 1. Quantitative comparison of different methods on the IndianPines dataset. Best results are highlighted in bold.

[0033] PaviaUniversity dataset: The present invention achieves OA=99.96%, AA=99.38%, and Kappa=99.80%, which is 0.43 percentage points higher than the suboptimal method HSI-MFormer (OA=99.53%). Table 2. Quantitative comparison of different methods on the PaviaUniversity dataset. Best results are highlighted in bold.

[0034] Salinas dataset: The present invention achieves OA=99.89%, AA=99.93%, and Kappa=98.45%, which is 0.37 percentage points higher than the suboptimal method HyperMamba (OA=99.52%). Table 3. Quantitative comparison of different methods on the Salinas dataset. Best results are highlighted in bold.

[0035] The contribution of each modality was quantified through ablation experiments, as shown in Figure 8(a). Figure 8(a) shows the t-SNE feature visualization results when only the SSIB module is used. The results demonstrate that the SSIB module can only achieve basic spatial-spectral feature coupling, and the feature discriminative ability is still significantly insufficient. The features in the figure show significant inter-class overlap, especially for land cover categories with high spectral similarity, where feature clusters intersect and the model has difficulty distinguishing them effectively. At the same time, some categories exhibit severe intra-class discrimination, and the features of the same category are loosely distributed in low-dimensional space. This indicates that spatial-spectral joint modeling using only 2D / 3D convolution cannot fully explore the deep discriminative features of hyperspectral images. However, as a basic module, it has solved the spatial-spectral separation problem of traditional single-modal modeling and is the basis for the design of subsequent modules.

[0036] As shown in Figure 8(b), the t-SNE feature visualization results using the SSIB+RG-FSEM module demonstrate that the RG-FSEM module achieves a key improvement in feature discriminativeness through frequency-aware modeling and RGB semantic space modulation. Previously overlapping feature clusters in the figure begin to separate clearly, significantly alleviating the class confusion problem caused by irregular spatial structure and spectral redundancy. The intra-class dispersion of each class is greatly reduced, and the feature distribution in the low-dimensional space is more compact. This indicates that the frequency grouping encoding and RGB-guided deformable convolution of RG-FSEM effectively filter redundant features and strengthen class-specific spatial-spectral joint features, verifying the necessity of frequency domain modeling and multimodal spatial guidance, and laying a high-quality feature foundation for subsequent long-distance inter-spectral dependency modeling.

[0037] As shown in Figure 8(c), the t-SNE feature visualization results are obtained using the complete SSIB+RG-FSEM+TMSM module. This result demonstrates that the TMSM module, as the core module, achieves ultimate optimization of feature representation through multi-directional inter-spectral dependency modeling and class-level semantic gating. In the figure, feature clusters of all categories form independent and non-overlapping distributions in low-dimensional space. Even subcategories with highly similar spectral features can form clear category boundaries. Simultaneously, feature points of the same category almost aggregate into dense cluster structures with no obvious discrete points. This indicates that the TMSM module fully exploits long-distance inter-spectral dependencies across bands and, combined with semantic gating, achieves semantic constraints on features. Ultimately, this allows the model to learn highly discriminative features that are highly compact within classes and completely separated between classes, verifying the synergy and irreplaceability of the three modules.

[0038] Ablation experiments: When using SSIB alone, the OA values ​​for the three datasets were 94.39% / 91.30% / 90.15% respectively; after adding RG-FSEM, the OA improved to 99.31% / 99.80% / 99.96%; after adding TMSM, the OA reached the optimal 99.96% / 99.96% / 99.89%, verifying the necessity and synergy of each module.

[0039] The present invention provides a hyperspectral image classification device based on frequency-semantic co-modulation, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the image classification method.

Claims

1. A hyperspectral image classification method based on frequency-semantic co-modulation, characterized in that, Includes the following steps: S1. Input a hyperspectral image, perform block processing on the hyperspectral image to obtain multiple sets of hyperspectral image sub-blocks, and construct the corresponding input text; S2. Feature extraction is performed on the hyperspectral image sub-blocks through spatial-spectral interaction blocks. The spatial-spectral interaction blocks construct a dual-branch structure through three-dimensional convolution and two-dimensional convolution to jointly model spatial information and spectral information, thereby obtaining spatial-spectral fusion features. S3. Using RGB-guided frequency-spatial embedding blocks, frequency-spatial co-modulation is performed on the spatial-spectral fusion features based on the RGB image to obtain multi-directional frequency-spatial enhancement features; S4. The Mamba module is guided by adaptive multi-scan text to perform semantic guidance and multi-directional state space modeling on the multi-directional frequency-space enhancement features based on the input text, so as to obtain the final classification features. S5. Perform hyperspectral image classification based on the final classification features, and construct a total loss function through semantic alignment loss and classification loss to complete model training and classification output.

2. The hyperspectral image classification method based on frequency-semantic co-modulation according to claim 1, characterized in that, In step S2, the feature extraction process of the spatial-spectral interaction block includes: For each input hyperspectral image sub-block, parallel feature extraction is performed using both 3D and 2D convolutional branches: 3D Convolution Branch: Extract spatial-spectral joint features by sequentially passing through 3×3×3 3D convolution, batch normalization, GELU activation function, 3×3×N 3D convolution, batch normalization, and GELU activation function; 2D convolution branch: Spatial dimension features are extracted sequentially through 3×3 2D convolution, batch normalization, and GELU activation function; The output features of the 3D convolutional branch and the 2D convolutional branch are fused by channel reweighting to obtain spatial-spectral fusion features.

3. The hyperspectral image classification method based on frequency-semantic co-modulation according to claim 1, characterized in that, In step S3, the modulation process of the RGB-guided frequency-space embedding block includes: S3.1 Perform a two-dimensional fast Fourier transform on the spatial-spectral fusion features to obtain frequency domain complex-valued features; decompose the frequency domain complex-valued features into real features and imaginary features; divide the spectral channels into several groups, and perform group-level learnable transformation and activation processing on the real features and imaginary features respectively; perform residual fusion on the processed real features and imaginary features and reassemble them into new frequency domain complex-valued features; perform a two-dimensional inverse fast Fourier transform on the new frequency domain complex-valued features to obtain frequency enhancement features; S3.2 For the corresponding RGB image, generate RGB guided modulation features by sequentially passing two layers of 3×3 convolution, batch normalization, and ReLU activation function; S3.3 Multiply the frequency enhancement features and RGB guided modulation features element by element, and then perform deformable convolution enhancement and root mean square normalization in sequence to obtain multi-directional frequency-space enhancement features.

4. The hyperspectral image classification method based on frequency-semantic co-modulation according to claim 1, characterized in that, In step S4, the process of self-adaptive multi-scan text bootstrapping of the Mamba module includes: S4.1 Perform 3D conditional position encoding on the input multi-directional frequency-spatial enhancement features and inject spatial spectral position information; normalize and linearly project the encoded features to obtain state-space modeling branch features and gated branch features, respectively. S4.

2. Convolution and activation are performed on the state space modeling branch features to enhance the local context. The text encoder is used to extract the category text embedding. Semantic gating vectors are generated through linear projection and activation. The state space modeling branch features are modulated element by element. Multi-directional selective state space scanning is performed on the modulated features to obtain multiple directional scanning features. S4.

3. The gated branch features are linearly projected and activated to generate adaptive fusion weights. The multiple directional scanning features are weighted and fused, and then linearly projected to output the final classification features.

5. The hyperspectral image classification method based on frequency-semantic co-modulation according to claim 1, characterized in that, In step S5, the formula for constructing the total loss function is: in, The classification loss is used to supervise the classification accuracy of hyperspectral images; λ is the semantic alignment loss, used to constrain the spatial-semantic consistency between text semantic features and visual features; λ is the balancing weight coefficient.

6. The hyperspectral image classification method based on frequency-semantic co-modulation according to claim 1, characterized in that, The input text is a category semantic description text corresponding to the hyperspectral image scene, used to provide semantic supervision guidance for visual feature extraction.

7. The hyperspectral image classification method based on frequency-semantic co-modulation according to claim 1, characterized in that, The model was trained using the AdamW optimizer with an initial learning rate of 1e-5 to 1e-3, a weight decay of 1e-3 to 1e-1, chord annealing for learning rate scheduling, a warm-up period of 3-8 epochs, a total training epoch of 40-60, and a batch size of 2-8.

8. The hyperspectral image classification method based on frequency-semantic co-modulation according to claim 4, characterized in that, The multi-directional selective state space scan includes at least two of the row direction, column direction, and diagonal direction.

9. A hyperspectral image classification device based on frequency-semantic co-modulation, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the hyperspectral image classification method as described in any one of claims 1-8.

Images