A multi-scale hyperspectral image classification method and device of frequency domain Mamba

Abstract

This invention provides a multi-scale hyperspectral image classification method and apparatus using frequency-domain Mamba, relating to the field of hyperspectral image classification technology. The method includes: collecting hyperspectral image data, preprocessing it, and segmenting it into multiple image patches; inputting the image patches into a spatial-spectral interaction feature extraction module, extracting interaction features through spatial branching and spectral-spatial fusion branches; inputting the interaction features into a frequency-domain perturbation embedding module to obtain frequency-domain processed features; inputting the frequency-domain processed features into an adaptive bidirectional Mamba module for forward and backward spatial and spectral feature extraction; and integrating the spectral and spatial features from different stages at multiple levels to obtain global features and achieve final classification. This invention can significantly improve the accuracy and robustness of hyperspectral image classification, and exhibits stronger generalization ability and stability under different datasets and complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral image classification technology, and in particular to a multi-scale hyperspectral image classification method and apparatus using frequency domain Mamba. Background Technology

[0002] Hyperspectral images are high-dimensional data composed of a large number of continuous spectral bands, providing rich spatial and spectral details. This characteristic has led to widespread interest in fields such as geological exploration, anomaly detection, precision agriculture, atmospheric science, and military reconnaissance. However, due to their complex spectral-spatial relationships, achieving high-precision hyperspectral image classification remains a challenging research topic.

[0003] Traditional hyperspectral image classification methods typically employ techniques such as support vector machines, sparse representation models, linear regression, and maximum a posteriori estimation. These methods primarily rely on shallow feature extraction, making it difficult to obtain discriminative and informative features in complex environments. For example, these methods often depend on single-scale feature processing, failing to effectively integrate information from different scales, leading to loss of detail. By neglecting the complementarity of different levels, contextual information is prone to inaccuracy, affecting image quality. The lack of hierarchical structure makes it difficult for the model to adjust feature importance, resulting in inaccurate information transmission and impacting classification performance.

[0004] In recent years, deep learning technology has significantly improved the performance of hyperspectral image classification, with convolutional neural networks (CNNs) being widely used due to their powerful feature extraction capabilities. However, the sliding window-based architecture of CNNs has limitations in capturing global context and long-range dependencies. With the development of attention mechanisms, Transformer-based models have performed excellently in hyperspectral image classification tasks, capturing dependencies between distant locations and spectral bands through global modeling capabilities. However, while Transformer-based self-attention mechanisms can effectively model global relationships, their quadratic computational complexity significantly increases model complexity, leading to increased computational overhead and resource consumption.

[0005] Against this backdrop, State-Space Models (SSMs) have emerged as an emerging approach, demonstrating a powerful ability to model long-range dependencies through state transition mechanisms. The Mamba model is a typical example of an SSM, exhibiting superior performance on multiple benchmark tasks through linear scalability while maintaining linear complexity. However, existing Mamba models still have limitations in hyperspectral image classification. For instance, flattening a two-dimensional image into a one-dimensional sequence destroys local spatial dependencies, leading to a loss of spatial information. Furthermore, single-scale spectral scanning limits the full utilization of spectral contextual information, particularly impacting classification accuracy when dealing with detailed features such as texture and boundaries. Summary of the Invention

[0006] To address the aforementioned problems, this invention aims to provide a multi-scale hyperspectral image classification method and apparatus using frequency-domain Mamba. By introducing a frequency-domain perturbation embedding module, it extracts high-frequency detail features and low-frequency global information using two-dimensional Fourier transform, and achieves efficient extraction of local and global features through grouped convolutional embedding, enhancing the comprehensive utilization of spatial information. An adaptive bidirectional Mamba module is designed, which effectively captures long-distance dependencies between spectral bands through bidirectional spectral analysis and remote modeling capabilities, and optimizes the balance between global information and local details by combining frequency-domain characteristics, thereby improving classification accuracy. A feature integration mechanism is proposed, achieving deep fusion of multi-layer features through adaptive cross-layer feature recalibration, effectively improving the problems of blurred classification boundaries and unclear region distinctions. Simultaneously, through hierarchical structure design, multi-scale spectral features are gradually extracted and spatial multi-scale features are integrated, significantly improving the generalization ability and performance of the classification model in complex scenarios.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] On the one hand, a multi-scale hyperspectral image classification method using frequency domain Mamba is provided, the method comprising the following steps:

[0009] S1. Collect hyperspectral image data and establish a sample dataset;

[0010] S2. Preprocess the sample dataset, divide it into training and test sets, and segment the input image into multiple image patches;

[0011] S3. Input the image block into the spatial spectral interactive feature extraction module, extract local spatial features through spatial branch, extract spectral spatial features through spectral spatial fusion branch, and combine the local spatial features with the spectral spatial features to obtain interactive features;

[0012] S4. Input the interactive features into the frequency domain perturbation embedding module, convert them to the frequency domain through fast Fourier transform, extract the frequency domain features after adding noise perturbation, and then restore them through inverse fast Fourier transform to obtain the frequency domain processed features.

[0013] S5. Input the frequency domain processed features into the adaptive bidirectional Mamba module to extract forward and backward spatial and spectral features;

[0014] S6. Use the spectral feature integration module to integrate the spectral features of different stages to obtain the integrated spectral features;

[0015] S7. Use the spatial feature integration module to integrate the spatial features of different stages to obtain the integrated spatial features;

[0016] S8. Further integrate the integrated spectral features and integrated spatial features to obtain global features and achieve final classification.

[0017] Optionally, step S1 specifically includes:

[0018] Hyperspectral image data were collected from publicly available hyperspectral image datasets to create a sample dataset;

[0019] The publicly available hyperspectral image datasets include those from Indian Pines, Pavia University, Trento, and LongKou.

[0020] Optionally, step S2 specifically includes:

[0021] The sample dataset is preprocessed, including: removing noisy bands, normalization, and data augmentation;

[0022] The preprocessed data was divided into training and testing sets in a 9:1 ratio.

[0023] Input image Where B represents the batch size, C in Let H be the number of spectral channels in the input image, and H and W be the height and width of the input image, respectively. Divide the input image into N image patches, denoted as Ni. ,in This represents the i-th image block.

[0024] Optionally, the spatial spectral interaction feature extraction module includes a spatial branch and a spectral spatial fusion branch, and step S3 specifically includes:

[0025] Segmented image blocks Simultaneously input the spatial branch and the spectral spatial fusion branch;

[0026] In the spatial branch, image patches are first processed through 1×1 convolutions to interact channels while maintaining spatial resolution, followed by 3×3 convolutions to capture local spatial features. Both types of convolutions are accompanied by batch normalization and the GELU activation function, as shown in the following formulas:

[0027]

[0028] Among them, Conv 2d Represents 2D convolution, BN represents batch normalization, GELU represents the activation function, and x represents 2D convolution. spa This represents the local spatial features extracted through the spatial branching process;

[0029] In the spectral spatial fusion branch, extraction is performed using a 1×1×1 3D convolution, as shown in the following formula:

[0030]

[0031] Among them, Conv 3d Represents 3D convolution, x spe This represents the spectral spatial features extracted through the spectral spatial fusion branch;

[0032] The local spatial features and the spectral spatial features are added element-wise and combined, and an attention map is generated through dynamic gating embedding. The combined features are then multiplied element-wise by the attention map, as shown in the following formula:

[0033]

[0034] Where, x fus σ represents the combined features, σ represents the activation function, and x represents the combined features. SSIB This represents the final interactive features obtained.

[0035] Optionally, step S4 specifically includes:

[0036] For interaction feature x SSIB Perform a Fast Fourier Transform:

[0037]

[0038] Where F is the Fast Fourier Transform operation, x SSIB Here, t represents the interactive features extracted by the spatial spectral interactive feature extraction module, and t is the feature map after Fast Fourier Transform. real For the real part features of the input, t imag The imaginary part of the input;

[0039]

[0040] t real N To represent the real part feature after adding noise perturbation, t imag N The imaginary part features after adding noise perturbation are defined as follows: α1 and α2 represent multiplication noise, which are sampled from the uniform distribution U(-1,1), and β1 and β2 represent addition noise, which are sampled from the normal distribution N(0,1).

[0041] The real and imaginary features after adding noise perturbation are subjected to group convolution embedding, followed by GELU nonlinear activation function processing:

[0042]

[0043] Conv g Represents group convolution. This represents the real part output after group convolution processing. This represents the imaginary part output after group convolution; then, information fusion is performed through complex number transformation:

[0044]

[0045] Among them, t out The output complex feature map consists of real and imaginary parts; finally, the frequency domain processed features are obtained through inverse fast Fourier transform.

[0046]

[0047] Among them, F -1 T represents the inverse fast Fourier transform. L These are the features after frequency domain processing.

[0048] Optionally, step S5 specifically includes:

[0049] Feature T after frequency domain processing L Rotational position encoding is performed to obtain the embedded sequence representation T. L ROPE The embedded sequence representation T L ROPE Normalize and linearly project to a high-dimensional space to obtain forward features Let z be the z-axis.

[0050]

[0051] forward features Reverse acquisition As a backward feature, then and As input, perform forward and backward spatial feature extraction; forward features and backward features The inputs are respectively processed by 1D convolution and the activation function SiLU, and then fed into the state-space equation SSM. The SSM models the transition from input x(t) to output y(t) through the hidden state h(t). The state-space equation SSM is defined as follows:

[0052]

[0053] Where A is the state transition matrix, B maps the input to the state, and C is the state-to-output transition; hyperspectral data is a discrete signal, and it is discretized using the zero-order preservation method, making the continuous state-space model applicable to discrete sequence modeling. The transformation formula is as follows:

[0054]

[0055] in, The parameters are discretized, with ΔA, ΔB, and ΔC being learnable parameters introduced by the Mamba structure to dynamically adjust the model's contextual representation and improve the adaptability of spectral features; x t It is the state at time t, h t It is the hidden state at time t, h t-1 Let I be the hidden state at time t-1, and let I be the unit feature vector.

[0056] After applying the state-space equation SSM, multiplying by the gated activation function G yields the output features F of the forward and backward branches. f and F b The specific process is as follows:

[0057]

[0058] in, The data flipping operation represents the output features of each branch, which are then processed by the average pooling layer (AP) and the convolutional layer (Conv) in subsequent stages. 2d The system first uses the ReLU activation function layer, and then uses the Sigmoid activation function to calculate the weights. Finally, the different features are weighted and fused to obtain the final output.

[0059]

[0060] Where W represents the weights calculated by the Sigmoid activation function, and F... fuse This indicates the characteristics after fusion.

[0061] Optionally, step S6 specifically includes:

[0062] Spectral features from three different stages are extracted and input into different convolutional layers C1, C2, and C3 respectively, and channel dimension alignment is performed.

[0063] After completing the channel dimension alignment, the spectral features are entered into the channel attention block, and the weights θ1, θ2, and θ3 of each channel are further adjusted.

[0064] The adjusted spectral features are integrated by element-wise addition to obtain the integrated spectral features.

[0065] Optionally, step S7 specifically includes:

[0066] Spatial features from three different stages were extracted and then upsampled by 8x, 4x and 2x respectively to construct cross-resolution feature representations.

[0067] The spatial features of the first stage are fused with the spatial features of the second and third stages after upsampling through symmetrical structures of 2x downsampling and 4x downsampling, respectively.

[0068] Each stage introduces a linear layer and a sigmoid activation function, followed by convolutional layers S1, S2, and S3. The weights of each stage are dynamically adjusted using learnable parameters δ1, δ2, and δ3.

[0069] The adjusted spatial features are integrated by element-wise addition to obtain the integrated spatial features.

[0070] Optionally, step S8 specifically includes:

[0071] The integrated spectral features and integrated spatial features are sequentially passed through a global average pooling layer and a fully connected layer to obtain global features;

[0072] The class probability of the global features is calculated using the Softmax classifier, and the classification result is output.

[0073] On the other hand, a frequency-domain Mamba multi-scale hyperspectral image classification apparatus is provided for implementing the method described in any of the above embodiments, the apparatus comprising:

[0074] The data collection module is used to collect hyperspectral image data and build a sample dataset;

[0075] The preprocessing module is used to preprocess the sample dataset, divide it into training and test sets, and segment the input image into multiple image patches;

[0076] The spatial-spectral interactive feature extraction module includes a spatial branch and a spectral-spatial fusion branch. For an input image patch, local spatial features are extracted through the spatial branch, spectral spatial features are extracted through the spectral-spatial fusion branch, and the local spatial features are combined with the spectral spatial features to obtain interactive features.

[0077] The frequency domain perturbation embedding module is used to convert the interactive features to the frequency domain through fast Fourier transform, extract the frequency domain features after adding noise perturbation, and then restore them through inverse fast Fourier transform to obtain the frequency domain processed features.

[0078] The adaptive bidirectional Mamba module is used to extract forward and backward spatial and spectral features from the features processed in the frequency domain.

[0079] The spectral feature integration module is used to integrate spectral features from different stages to obtain integrated spectral features;

[0080] The spatial feature integration module is used to integrate spatial features from different stages to obtain integrated spatial features;

[0081] The global feature integration and classification module is used to further integrate the integrated spectral features and the integrated spatial features to obtain global features and achieve final classification.

[0082] On the other hand, an electronic device is provided, the electronic device comprising:

[0083] processor;

[0084] A memory storing computer-readable instructions, which, when loaded and executed by the processor, implement the steps of the multi-scale hyperspectral image classification method of frequency domain Mamba as described above.

[0085] On the other hand, a computer-readable storage medium is provided, wherein program code is stored in the computer-readable storage medium, the program code being invoked by a processor to perform the steps of the multi-scale hyperspectral image classification method of frequency domain Mamba described above.

[0086] The beneficial effects of the technical solution provided by this invention include at least the following:

[0087] (1) This invention effectively solves the problem of insufficient spatial information extraction in hyperspectral image classification by using a spatial spectral interactive feature extraction module. It uses multi-scale convolution and 3D convolution to extract local and global features, and combines a gating enhancement mechanism to optimize feature expression, thereby improving the fusion capability of spatial and spectral features and thus enhancing classification accuracy.

[0088] (2) This invention innovatively introduces a frequency domain perturbation embedding module, which uses two-dimensional Fourier transform to extract high-frequency detail features and low-frequency global information. By increasing noise perturbation, the feature representation ability is improved, and combined with group convolution to optimize feature extraction, the model can more accurately separate edge details and global background information, thereby improving the clarity of classification boundaries and the ability to distinguish categories.

[0089] (3) In view of the limitations of existing methods in modeling long-range dependencies and insufficient utilization of spectral information, this invention designs an adaptive bidirectional Mamba module, which improves the dependency modeling capability of spectral dimension through bidirectional spectral analysis mechanism and rotation position encoding, ensuring that the model can more comprehensively understand the spectral information of hyperspectral data and improve the robustness of classification.

[0090] (4) This invention adopts a spectral feature and spatial feature integration mechanism. Through cross-layer feature recalibration, it achieves deep fusion of spectral features and spatial features at different levels. Combined with hierarchical architecture design, it gradually extracts multi-layer features, so that the model can show stronger generalization ability and stability in different datasets and complex scenarios, thereby effectively improving the accuracy and reliability of hyperspectral image classification. Attached Figure Description

[0091] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0092] Figure 1 This is a general block diagram of a multi-scale hyperspectral image classification method using frequency domain Mamba provided in an embodiment of the present invention;

[0093] Figure 2 This is a schematic diagram of image preprocessing provided in an embodiment of the present invention;

[0094] Figure 3 This is a schematic diagram of the spatial spectral interactive feature extraction module provided in an embodiment of the present invention;

[0095] Figure 4 This is a schematic diagram of the frequency domain perturbation embedding module provided in an embodiment of the present invention;

[0096] Figure 5 This is a schematic diagram of the adaptive bidirectional Mamba module provided in an embodiment of the present invention;

[0097] Figure 6 This is a schematic diagram of the spectral feature integration module provided in an embodiment of the present invention;

[0098] Figure 7 This is a schematic diagram of the spatial feature integration module provided in an embodiment of the present invention;

[0099] Figure 8 This is a schematic diagram of global information integration and classification provided in an embodiment of the present invention. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0101] In embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the term "exemplary" is intended to present the concept in a specific manner.

[0102] This invention provides a multi-scale hyperspectral image classification method using frequency-domain Mamba, which can be implemented by an electronic device, such as a terminal or a server. Figure 1 As shown, the processing flow of this method includes the following steps:

[0103] S1. Collect hyperspectral image data and establish a sample dataset.

[0104] In this embodiment of the invention, hyperspectral image data is collected from publicly available hyperspectral image datasets to establish a sample dataset. These publicly available hyperspectral image datasets include those from Indian Pines, Pavia University, Trento, and LongKou, and contain rich spectral and spatial information.

[0105] S2. Preprocess the sample dataset, divide it into training and test sets, and segment the input image into multiple image patches.

[0106] like Figure 2 As shown, the preprocessing of the sample dataset includes: removing noisy bands, normalization, and data augmentation. Afterward, the preprocessed data is divided into training and testing sets in a 9:1 ratio to ensure data quality and the model's generalization ability.

[0107] The preprocessed image is used as the input image of the model, denoted as... Where B represents the batch size, C in Let H be the number of spectral channels in the input image, and H and W be the height and width of the input image, respectively. The input image is then divided into N image patches, denoted as Ni. ,in This represents the i-th image block.

[0108] S3. Input the image patch into the spatial spectral interactive feature extraction module, extract local spatial features through spatial branch, extract spectral spatial features through spectral spatial fusion branch, and combine the local spatial features with the spectral spatial features to obtain interactive features.

[0109] like Figure 3 As shown, the spatial spectral interactive feature extraction module includes a spatial branch and a spectral spatial fusion branch, which extracts segmented image patches. Simultaneously input spatial branch and spectral spatial fusion branch.

[0110] In the spatial branch, image patches are first processed through 1×1 convolutions to interact channels while maintaining spatial resolution, followed by 3×3 convolutions to capture local spatial features. Both types of convolutions are accompanied by batch normalization and the GELU activation function to promote stable training and introduce non-linearity, as shown in the following formula:

[0111]

[0112] Among them, Conv 2d Represents 2D convolution, BN represents batch normalization, GELU represents the activation function, and x represents 2D convolution. spa This represents the local spatial features extracted through the spatial branch.

[0113] In the spectral spatial fusion branch, extraction is performed using a 1×1×1 3D convolution, as shown in the following formula:

[0114]

[0115] Among them, Conv 3d Represents 3D convolution, x spe This represents the spectral spatial features extracted through the spectral spatial fusion branch.

[0116] The local spatial features and spectral spatial features are added element-wise and combined, and then an attention map is generated through dynamic gating embedding. The combined features are then multiplied element-wise by this attention map, as shown in the following formula:

[0117]

[0118] Where, x fus σ represents the combined features, σ represents the activation function, and x represents the combined features. SSIB This represents the final interactive features obtained.

[0119] S4. Input the interactive features into the frequency domain perturbation embedding module, convert them to the frequency domain through fast Fourier transform, extract the frequency domain features after adding noise perturbation, and then restore them through inverse fast Fourier transform to obtain the frequency domain processed features.

[0120] like Figure 4 As shown, the frequency domain perturbation embedding module uses a two-dimensional fast Fourier transform to convert the input image to the frequency domain to enhance feature separability. Low-frequency components retain global information, while high-frequency components extract local texture information. Noise perturbations are added to both high and low frequencies to improve the ability to recognize boundary regions. Group convolutions are used to reduce computational complexity and optimize the feature representation capabilities of different frequency domain components. By combining frequency domain feature extraction and removing redundant information, the model can learn more discriminative features.

[0121] The Fourier transform is used to convert time-domain signals to the frequency domain, helping to analyze the frequency domain characteristics of hyperspectral images. The mathematical representation of the two-dimensional discrete Fourier transform is as follows:

[0122]

[0123] Where F(h,w) represents the signal value at position (h,w), and u and v are the horizontal and vertical spatial frequency indices in the frequency domain, respectively. The corresponding inverse Fourier transform is used to recover the time-domain information:

[0124]

[0125] In this embodiment of the invention, the interaction feature x SSIB Perform a Fast Fourier Transform:

[0126]

[0127]

[0128] Where F is the Fast Fourier Transform operation, x SSIB Here, t represents the interactive features extracted by the spatial spectral interactive feature extraction module, and t is the feature map after Fast Fourier Transform. real For the real part features of the input, t imag The imaginary part of the input is used for feature definition.

[0129]

[0130] t real N To represent the real part feature after adding noise perturbation, t imag N The imaginary part features after adding noise perturbation are defined as follows: α1 and α2 represent multiplication noise, which are sampled from the uniform distribution U(-1,1), and β1 and β2 represent addition noise, which are sampled from the normal distribution N(0,1).

[0131] Group convolution embedding is applied to the real and imaginary features after noise perturbation, which effectively integrates spectral channel information and reduces complexity. Then, GELU nonlinear activation function is applied to enhance the nonlinear feature representation.

[0132]

[0133] Conv g Represents group convolution. This represents the real part output after group convolution processing. This represents the imaginary part output after group convolution. Next, information fusion is performed through complex number transformations:

[0134]

[0135] Among them, t out The output complex feature map consists of real and imaginary parts. Finally, the frequency domain features are obtained through inverse fast Fourier transform:

[0136]

[0137] Among them, F -1 T represents the inverse fast Fourier transform. L These are the features after frequency domain processing.

[0138] In order to process according to a hierarchical structure, the obtained T L The spectral dimension is mapped to a higher feature space. This module is designed to ensure that the model can efficiently extract feature information in the frequency domain and reconstruct optimized data in the spatial domain, thereby improving the model's ability to classify hyperspectral images.

[0139] S5. Input the frequency domain processed features into the adaptive bidirectional Mamba module to extract forward and backward spatial and spectral features.

[0140] like Figure 5 As shown, the feature T after frequency domain processing L Rotational position encoding is performed to obtain the embedded sequence representation T. L ROPE The embedded sequence representation T L ROPE Normalize and linearly project to a high-dimensional space to obtain forward features Let z be the z-axis.

[0141]

[0142] forward features Reverse acquisition As a backward feature, then and As input, perform forward and backward spatial feature extraction. Forward features and backward features The data are processed using 1D convolution and the SiLU activation function, respectively, and then input into the state-space equation SSM. This module uses the state-space model SSM to process spectral sequence data, describing the state evolution with a linear ordinary differential equation (ODE). SSM models the transition from input x(t) to output y(t) through the hidden state h(t). The state-space equation SSM is defined as follows:

[0143]

[0144] Where A is the state transition matrix, B maps the input to the state, and C is the state-to-output transition; hyperspectral data is a discrete signal, and it is discretized using the zero-order preservation method, making the continuous state-space model applicable to discrete sequence modeling. The transformation formula is as follows:

[0145]

[0146] in, The parameters are discretized, and ΔA, ΔB, and ΔC are learnable parameters introduced by the Mamba structure to dynamically adjust the contextual representation of the model and improve the adaptability of spectral features. t It is the state at time t, h t It is the hidden state at time t, h t-1 I is the hidden state at time t-1, and I is the unit eigenvector.

[0147] After applying the state-space equation SSM, multiplying by the gated activation function G yields the output features F of the forward and backward branches. f and F b The specific process is as follows:

[0148]

[0149] in, This represents the data flipping operation. The output features of each branch are then processed in subsequent stages by the average pooling layer (AP) and the convolutional layer (Conv). 2d The ReLU activation function layer is used. First, global features are integrated through an average pooling layer. Then, spectral information is extracted through multiple convolutional layers and subjected to a nonlinear transformation through the ReLU activation function layer to enhance the expressive power of the features.

[0150] Finally, after calculating the weights using the Sigmoid activation function, the different features are weighted and fused to obtain the final output:

[0151]

[0152] Where W represents the weights calculated by the Sigmoid activation function, and F... fuse This module represents the fused features. Through multi-branch convolution, state modeling, and feature fusion, the model adaptively captures spectral information, improving the discriminative power of the final features.

[0153] S6. Use the spectral feature integration module to integrate the spectral features of different stages to obtain the integrated spectral features.

[0154] like Figure 6As shown, this module processes the spectral features of three stages, extracts the spectral features of different stages, and aligns them. Specifically, it includes: extracting the spectral features of three different stages and inputting them into different convolutional layers C1, C2, and C3 respectively, performing channel dimension alignment; after completing the channel dimension alignment, the spectral features enter the channel attention blocks, further adjusting the weights θ1, θ2, and θ3 of each channel to enhance the model's attention to important features; the adjusted spectral features are integrated through element-wise addition to achieve effective feature fusion and refinement, finally obtaining the integrated spectral features.

[0155] S7. Use the spatial feature integration module to integrate the spatial features of different stages to obtain the integrated spatial features.

[0156] like Figure 7 As shown, this module employs a multi-scale spatial feature interaction mechanism, achieving deep integration of spatial context information through hierarchical sampling and nonlinear transformation. Specifically, it includes: extracting spatial features from three different stages, which are then upsampled by 8x, 4x, and 2x to construct cross-resolution feature representations; subsequently, the spatial features from the first stage are fused with the upsampled spatial features from the second and third stages through symmetrical structures of 2x and 4x downsampling, respectively, further strengthening the coupling between local and global spatial features; each stage introduces a linear layer and a sigmoid activation function to perform dimensionality compression and nonlinear calibration on the features, respectively, followed by convolutional layers S1, S2, and S3 to extract spatially sensitive feature responses. The weights of each stage are dynamically adjusted using learnable parameters δ1, δ2, and δ3 to adaptively balance the semantic differences of multi-scale features; the adjusted spatial features are then integrated through element-wise addition to obtain the integrated spatial features.

[0157] In this embodiment of the invention, through cascaded multi-stage processing, the module can effectively capture spatial dependencies under different receptive fields and suppress redundant information during feature fusion, ultimately outputting spatially enhanced features with strong discriminative power. This design significantly improves the model's ability to model complex spatial structures, especially in terms of texture detail restoration and geometric consistency preservation.

[0158] S8. Further integrate the integrated spectral features and integrated spatial features to obtain global features and achieve final classification.

[0159] like Figure 8 As shown, the integrated spectral features and integrated spatial features are sequentially passed through a global average pooling layer and a fully connected layer to obtain global features; then, a Softmax classifier is used to calculate the class probability of the global features and output the classification result.

[0160] Specifically, global average pooling is used to compress spatial information, reducing computational burden and improving model inference speed. Fully connected layers are used for final classification, and softmax is combined to calculate class probabilities, achieving efficient classification decisions. The goal of this step is to ensure that the classifier maintains stable performance across different scenarios and datasets.

[0161] Accordingly, embodiments of the present invention also provide a frequency-domain Mamba multi-scale hyperspectral image classification apparatus, the apparatus being used to implement the above-described multi-scale hyperspectral image classification method, the apparatus comprising:

[0162] The data collection module is used to collect hyperspectral image data and build a sample dataset;

[0163] The preprocessing module is used to preprocess the sample dataset, divide it into training and test sets, and segment the input image into multiple image patches;

[0164] The spatial-spectral interactive feature extraction module includes a spatial branch and a spectral-spatial fusion branch. For an input image patch, local spatial features are extracted through the spatial branch, spectral spatial features are extracted through the spectral-spatial fusion branch, and the local spatial features are combined with the spectral spatial features to obtain interactive features.

[0165] The frequency domain perturbation embedding module is used to convert the interactive features to the frequency domain through fast Fourier transform, extract the frequency domain features after adding noise perturbation, and then restore them through inverse fast Fourier transform to obtain the frequency domain processed features.

[0166] The adaptive bidirectional Mamba module is used to extract forward and backward spatial and spectral features from the features processed in the frequency domain.

[0167] The spectral feature integration module is used to integrate spectral features from different stages to obtain integrated spectral features;

[0168] The spatial feature integration module is used to integrate spatial features from different stages to obtain integrated spatial features;

[0169] The global feature integration and classification module is used to further integrate the integrated spectral features and the integrated spatial features to obtain global features and achieve final classification.

[0170] The apparatus of this embodiment can be used to perform Figure 1 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0171] Compared with the prior art, the present invention has the following advantages:

[0172] (1) Addressing the problem of insufficient spatial information extraction: Existing methods process images by flattening them into one-dimensional sequences, resulting in the loss of spatial information. To address this issue, this invention designs a spatial-spectral interaction module that effectively fuses spatial and spectral information. This module utilizes two-dimensional convolution to extract spatial features from the image, while simultaneously using three-dimensional convolution to interact with spectral and spatial information, preserving the spatial local dependencies in the image and enhancing the ability to capture spatial information, thereby improving classification accuracy. A frequency domain perturbation embedding module is also introduced, which converts the data to the frequency domain using a two-dimensional fast Fourier transform, extracting high-frequency details and low-frequency global information respectively, and enhancing feature representation capabilities by adding noise perturbation. Group convolution operations are used to extract features in different frequency domains, enabling information fusion at the frequency domain level, effectively capturing detailed features and global structures in the image, reducing the problem of blurred regional boundaries, and improving the accuracy of image classification.

[0173] (2) Addressing the issues of long-range dependencies and insufficient utilization of spectral information: Traditional convolutional neural networks struggle to capture global context and long-range dependencies through sliding window mechanisms. While Transformer-based self-attention mechanisms can effectively model long-range dependencies, they suffer from high computational complexity and resource consumption. Therefore, this invention introduces an adaptive bidirectional Mamba module, which captures long-range dependencies by bidirectionally scanning spectral information. Combined with the linear state-space modeling capability of the Mamba model, computational efficiency is effectively improved. Mamba's linear transition characteristics result in lower computational costs when handling long-range dependencies, avoiding the high computational overhead of traditional methods while retaining sensitivity to local details. This improves the model's processing efficiency on large-scale data and its ability to capture global information.

[0174] (3) Addressing the problem of insufficient generalization ability: Existing methods typically rely on a single feature processing approach, lacking effective hierarchical processing, leading to insufficient information integration, especially in complex data scenarios, resulting in unstable classification performance. To address this issue, this invention adopts a hierarchical structure. This structure processes images layer by layer through multiple feature extraction stages, extracting feature information at different levels at each level, and fusing and weighting the features at different levels through spectral and spatial feature integration modules. Through this hierarchical processing, the model can adjust the importance of features at different levels, ensuring effective integration of multi-scale information and improving the model's generalization ability and classification accuracy.

[0175] In summary, this invention overcomes the limitations of traditional hyperspectral image classification methods by combining spatial spectral interactive feature extraction, frequency domain perturbation embedding, adaptive bidirectional Mamba, and hierarchical multi-scale feature fusion, achieving efficient and accurate hyperspectral image classification. After inputting hyperspectral image data, the spatial spectral interactive feature extraction module extracts interactive features, then the frequency domain perturbation embedding module and the adaptive bidirectional Mamba module model long-range dependencies, and finally, the hierarchical feature fusion module optimizes the classification results, outputting a high-precision classification map. Experimental results demonstrate that this method achieves leading classification accuracy on multiple hyperspectral datasets and outperforms existing methods in terms of boundary sharpness, computational efficiency, and generalization ability. By further optimizing computational efficiency, this invention can be applied to larger-scale hyperspectral data processing tasks and extended to the fields of multimodal data fusion and remote sensing applications.

[0176] In an exemplary embodiment, the present invention also provides an electronic device, the electronic device comprising:

[0177] processor;

[0178] A memory storing computer-readable instructions, which, when loaded and executed by the processor, implement the steps of the multi-scale hyperspectral image classification method of frequency domain Mamba as described above.

[0179] In an exemplary embodiment, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the steps of the multi-scale hyperspectral image classification method of frequency-domain Mamba described above. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0180] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0181] The use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0182] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0183] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0184] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0185] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0186] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0188] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-scale hyperspectral image classification method using frequency domain Mamba, characterized in that, Includes the following steps: S1. Collect hyperspectral image data and establish a sample dataset; S2. Preprocess the sample dataset, divide it into training and test sets, and segment the input image into multiple image patches; S3. Input the image block into the spatial spectral interactive feature extraction module. The spatial spectral interactive feature extraction module includes a spatial branch and a spectral spatial fusion branch. Local spatial features are extracted through the spatial branch, spectral spatial features are extracted through the spectral spatial fusion branch, and the local spatial features are combined with the spectral spatial features to obtain interactive features. Step S3 specifically includes: The segmented image blocks are simultaneously input into the spatial branch and the spectral spatial fusion branch; In the spatial branch, image patches are first processed by 1×1 convolution to interact channels while maintaining spatial resolution, and then 3×3 convolution is used to capture local spatial features; both types of convolution are accompanied by batch normalization and the GELU activation function. In the spectral spatial fusion branch, spectral spatial features are extracted through 1×1×1 3D convolution; The local spatial features and the spectral spatial features are added element by element and combined, and an attention map is generated by dynamic gating embedding. The combined features are then multiplied element by element with the attention map to obtain the interactive features. S4. Input the interactive features into the frequency domain perturbation embedding module, convert them to the frequency domain through fast Fourier transform, extract the frequency domain features after adding noise perturbation, and then restore them through inverse fast Fourier transform to obtain the frequency domain processed features. Step S4 specifically includes: The interactive features are subjected to Fast Fourier Transform; the real and imaginary features after noise perturbation are subjected to group convolution embedding, followed by GELU nonlinear activation function processing; then, information fusion is performed through complex transformation; finally, the frequency domain processed features are obtained through inverse Fast Fourier Transform. S5. Input the frequency domain processed features into the adaptive bidirectional Mamba module to extract forward and backward spatial and spectral features; Step S5 specifically includes: Rotational position encoding is performed on the frequency domain processed features to obtain the embedded sequence representation. The embedded sequence representation is then normalized and linearly projected onto a high-dimensional space to obtain the forward features. The forward features are inverted to obtain the backward features. Then, the forward and backward features are used as inputs to extract the forward and backward spatial features. The forward and backward features are respectively passed through 1D convolution and activation function SiLU, and then input into the state space equation SSM. SSM models the transformation from input to output through hidden states. After applying the state-space equation SSM, the output features of the forward and backward branches are obtained by multiplying them by the gated activation function G. The output features of each branch are then processed in subsequent stages by the average pooling layer AP and the convolutional layer Conv. 2d The ReLU activation function layer is used, and finally the weights are calculated by the Sigmoid activation function to perform weighted fusion of different features. S6. Use the spectral feature integration module to integrate the spectral features of different stages to obtain the integrated spectral features; Step S6 specifically includes: Spectral features from three different stages are extracted and input into different convolutional layers C1, C2, and C3 respectively, and channel dimension alignment is performed. After completing the channel dimension alignment, the spectral features are entered into the channel attention block, and the weights θ1, θ2, and θ3 of each channel are further adjusted. The adjusted spectral features are integrated by element-wise addition to obtain the integrated spectral features. S7. Use the spatial feature integration module to integrate the spatial features of different stages to obtain the integrated spatial features; Step S7 specifically includes: Spatial features from three different stages were extracted and then upsampled by 8x, 4x and 2x respectively to construct cross-resolution feature representations. The spatial features of the first stage are fused with the spatial features of the second and third stages after upsampling through symmetrical structures of 2x downsampling and 4x downsampling, respectively. Each stage introduces a linear layer and a sigmoid activation function, followed by convolutional layers S1, S2, and S3. The weights of each stage are dynamically adjusted using learnable parameters δ1, δ2, and δ3. The adjusted spatial features are integrated by element-wise addition to obtain the integrated spatial features. S8. Further integrate the integrated spectral features and integrated spatial features to obtain global features and achieve final classification.

2. The multi-scale hyperspectral image classification method according to claim 1, characterized in that, Step S1 specifically includes: Hyperspectral image data were collected from publicly available hyperspectral image datasets to create a sample dataset; The publicly available hyperspectral image datasets include those from Indian Pines, Pavia University, Trento, and LongKou.

3. The multi-scale hyperspectral image classification method according to claim 1, characterized in that, Step S2 specifically includes: The sample dataset is preprocessed, including: removing noisy bands, normalization, and data augmentation; The preprocessed data was divided into training and testing sets in a 9:1 ratio. Input image Where B represents the batch size, C in Let H be the number of spectral channels in the input image, and H and W be the height and width of the input image, respectively. Divide the input image into N image patches, denoted as Ni. in This represents the i-th image block.

4. The multi-scale hyperspectral image classification method according to claim 1, characterized in that, In step S3, the segmented image blocks Simultaneously input the spatial branch and the spectral spatial fusion branch; In the spatial branch, image patches are first processed through 1×1 convolutions to interact channels while maintaining spatial resolution, followed by 3×3 convolutions to capture local spatial features. Both types of convolutions are accompanied by batch normalization and the GELU activation function, as shown in the following formulas: Among them, Conv 2d Represents 2D convolution, BN represents batch normalization, GELU represents the activation function, and x represents 2D convolution. spa This represents the local spatial features extracted through the spatial branching process; In the spectral spatial fusion branch, extraction is performed using a 1×1×1 3D convolution, as shown in the following formula: Among them, Conv 3d Represents 3D convolution, x spe This represents the spectral spatial features extracted through the spectral spatial fusion branch; The local spatial features and the spectral spatial features are added element-wise and combined, and an attention map is generated through dynamic gating embedding. The combined features are then multiplied element-wise by the attention map, as shown in the following formula: x fus =x spa +x spe x SSIB =x fus ·σ(Conv 2d (x fus )) Where, x fus σ represents the combined features, σ represents the activation function, and x represents the combined features. SSIB This represents the final interactive features obtained.

5. The multi-scale hyperspectral image classification method according to claim 1, characterized in that, In step S4, the interaction feature x SSIB Perform a Fast Fourier Transform: t=F(x SSIB ) t real ,t imag =t Where F is the Fast Fourier Transform operation, x SSIB Here, t represents the interactive features extracted by the spatial spectral interactive feature extraction module, and t is the feature map after Fast Fourier Transform. real For the real part features of the input, t imag The imaginary part of the input; t real N =α1·t real +β1 t imag N =α2·t imag +β2 t real N To represent the real part of the feature after adding noise perturbation, t imag N The imaginary part features after adding noise perturbation are defined as follows: α1 and α2 represent multiplication noise, which are sampled from the uniform distribution U(-1,1), and β1 and β2 represent addition noise, which are sampled from the normal distribution N(0,1). The real and imaginary features after adding noise perturbation are subjected to group convolution embedding, followed by GELU nonlinear activation function processing: t' real =GELU(Conv g (t real N ))+t real t' imag =GELU(Conv g (t imag N ))+t imag Conv g Denotes group convolution, t′ real t′ represents the real part output after group convolution. imag This represents the imaginary part output after group convolution; then, information fusion is performed through complex number transformation: t out =Complex(Stack(t' real ,t' imag )) Among them, t out The output complex feature map consists of real and imaginary parts; finally, the frequency domain processed features are obtained through inverse fast Fourier transform. T L =F -1 (t+t out ) Among them, F -1 T represents the inverse fast Fourier transform. L These are the features after frequency domain processing.

6. The multi-scale hyperspectral image classification method according to claim 1, characterized in that, In step S5, the frequency domain processed feature T L Rotational position encoding is performed to obtain the embedded sequence representation T. L ROPE The embedded sequence representation T L ROPE Normalize and linearly project to a high-dimensional space to obtain forward features Let z be the z-axis. forward features Reverse acquisition As a backward feature, then and As input, perform forward and backward spatial feature extraction; Forward feature and backward features The inputs are respectively processed by 1D convolution and the activation function SiLU, and then fed into the state-space equation SSM. The SSM models the transition from input x(t) to output y(t) through the hidden state h(t). The state-space equation SSM is defined as follows: h'(t) = Ah(t) + Bx(t) y(t)=Ch(t) Where A is the state transition matrix, B maps the input to the state, and C is the state-to-output transition; hyperspectral data is a discrete signal, and it is discretized using the zero-order preservation method, making the continuous state-space model applicable to discrete sequence modeling. The transformation formula is as follows: in, The parameters are discretized, with ΔA, ΔB, and ΔC being learnable parameters introduced by the Mamba structure to dynamically adjust the model's contextual representation and improve the adaptability of spectral features; x t It is the state at time t, h t It is the hidden state at time t, h t-1 Let I be the hidden state at time t-1, and let I be the unit feature vector. After applying the state-space equation SSM, multiplying by the gated activation function G yields the output features F of the forward and backward branches. f and F b The specific process is as follows: G = SiLU(z) Here, Filp(·) represents the data flipping operation; the output features of each branch are processed in subsequent stages by the average pooling layer AP and the convolutional layer Conv. 2d The system first uses the ReLU activation function layer, and then uses the Sigmoid activation function to calculate the weights. Finally, the different features are weighted and fused to obtain the final output. W=Sigmoid(Conv 2d (RELU(Conv 2d (AP(F f +F b ))))) F fuse =Conv 2d (F f ·W+F b ·(1-W)) Where W represents the weights calculated by the Sigmoid activation function, and F... fuse This indicates the characteristics after fusion.

7. The multi-scale hyperspectral image classification method according to claim 1, characterized in that, Step S8 specifically includes: The integrated spectral features and integrated spatial features are sequentially passed through a global average pooling layer and a fully connected layer to obtain global features; The class probability of the global features is calculated using the Softmax classifier, and the classification result is output.

8. A frequency-domain Mamba multi-scale hyperspectral image classification device, the device being used to implement the method as described in any one of claims 1 to 7, characterized in that, The device includes: The data collection module is used to collect hyperspectral image data and build a sample dataset; The preprocessing module is used to preprocess the sample dataset, divide it into training and test sets, and segment the input image into multiple image patches; The spatial-spectral interactive feature extraction module includes a spatial branch and a spectral-spatial fusion branch. For an input image patch, local spatial features are extracted through the spatial branch, spectral spatial features are extracted through the spectral-spatial fusion branch, and the local spatial features are combined with the spectral spatial features to obtain interactive features. The frequency domain perturbation embedding module is used to convert the interactive features to the frequency domain through fast Fourier transform, extract the frequency domain features after adding noise perturbation, and then restore them through inverse fast Fourier transform to obtain the frequency domain processed features. The adaptive bidirectional Mamba module is used to extract forward and backward spatial and spectral features from the features processed in the frequency domain. The spectral feature integration module is used to integrate spectral features from different stages to obtain integrated spectral features; The spatial feature integration module is used to integrate spatial features from different stages to obtain integrated spatial features; The global feature integration and classification module is used to further integrate the integrated spectral features and the integrated spatial features to obtain global features and achieve final classification.

Images