Multi-source multi-mode remote sensing image classification method based on Mama spiral scanning and knowledge distillation

Abstract

The invention discloses a multi-source and multi-mode remote sensing image classification method based on Mama spiral scanning and knowledge distillation, and the method comprises the steps: obtaining multi-source and multi-mode remote sensing data which comprises a hyperspectral image, a synthetic aperture radar image and a laser radar image; the multi-source and multi-mode remote sensing data are input into an SMKT-Net network, the SMKT-Net network carries out spiral scanning processing on the hyperspectral image to obtain hyperspectral image features, cross-mode attention interaction is carried out on the hyperspectral image features and spatial features of the synthetic aperture radar image or the laser radar image to obtain fusion features, and the fusion features are fused with the hyperspectral image features and the spatial features of the synthetic aperture radar image or the laser radar image. The SMKT-Net network is obtained through training of a knowledge distillation training strategy; and inputting the fusion features into a classifier to obtain the probability of the target category. According to the method, the robustness in multi-source and multi-mode remote sensing scene classification and the applicability to complex terrain differences can be improved.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation. Background Technology

[0002] Remote sensing imagery has been widely applied in tasks such as land cover classification, environmental monitoring, precision agriculture assessment, and geological exploration, and its performance has been continuously proven. However, limited by the performance of the sensors themselves and the reflectivity of ground features, each sensor's images have their unique advantages and limitations. For example, hyperspectral images (HSI) can capture rich information about land cover by providing hundreds or even thousands of consecutive spectral channels. However, these images are susceptible to interference from changes in atmospheric conditions and dense vegetation cover, resulting in less than ideal image quality. In contrast, thanks to their independence from weather and lighting conditions, synthetic aperture radar (SAR) images can provide detailed information on surface roughness, topography, structure, and material properties, while light detection and ranging (LiDAR) images can provide high-precision surface elevation information, serving as an effective supplement to hyperspectral information. Therefore, in downstream remote sensing image analysis tasks requiring high-precision positioning and fine-grained identification, the shortcomings of each sensor are often compensated by fully combining multimodal remote sensing images from different sources. For example, SAR / LiDAR data can compensate for hyperspectral classification tasks under adverse weather conditions, while the rich spectral information in hyperspectral data can enhance the classification capabilities of SAR / LiDAR data. However, there are challenges in acquiring and realizing high-precision cross-modal spatial detail and rich spectral information representation and dynamic fusion from these multimodal remote sensing images.

[0003] Currently, most deep multimodal remote sensing scene classification algorithms focus on mining data features from different modalities, such as spectral attributes and spatial data, by designing various network structures. However, while existing work focuses on intermodal consistency, it neglects the dynamic interaction of cross-modal features between multi-source multimodal remote sensing data, such as feature concatenation and simple overlay of modal features. Therefore, further exploring this cross-modal dynamic interaction and using it to improve remote sensing scene classification performance is one of the problems that needs to be solved. In addition, some existing deep cross-modal classification optimization algorithms based on information entropy, such as the deep mutual information maximization for cross-modal clustering (DMIM) algorithm, only emphasize how to use mutual information to eliminate redundant information within a modality. However, in reality, due to the lack of guidance on clustering different target categories, DMIM has difficulty defining the boundary between redundant and effective information when classifying multi-source multimodal remote sensing scenes, and it ignores the redundancy between different modalities in multi-source multimodal data, i.e., the redundancy distribution within and between modalities, which limits classification performance.

[0004] Therefore, this invention provides a multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation, which improves the robustness of multi-source, multi-modal remote sensing scene classification and its applicability to complex terrain differences.

[0006] To achieve the above objectives, the present invention provides the following solution: A multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation includes: Acquire multi-source, multi-modal remote sensing data, wherein the multi-source, multi-modal remote sensing data includes hyperspectral images, synthetic aperture radar images, and lidar images; The multi-source multimodal remote sensing data is input into the SMKT-Net network. The SMKT-Net network performs helical scanning processing on the hyperspectral image to obtain hyperspectral image features. The hyperspectral image features and the spatial features of the synthetic aperture radar image or lidar image are subjected to cross-modal attention interaction to obtain fused features. The SMKT-Net network is trained through a knowledge distillation training strategy. The fused features are input into a classifier to obtain the probability of the target category.

[0007] Optionally, performing helical scanning processing on the hyperspectral image includes: Acquire the target pixel in the hyperspectral image, and extract the local image patch around the target pixel; The local image patch is spiral-scanned to obtain a spectral vector sequence; The spectral vector sequence is processed using a selective state-space model to aggregate spectral-spatial information and obtain the final hidden state, i.e., the hyperspectral image features. The final hidden state contains all the spectral-spatial features of the neighborhood of the central pixel.

[0008] Optionally, performing a spiral scan on the local image block includes: scanning the local image block spirally inward from the first pixel to the center pixel, arranging the spectral vectors of the pixels into a sequence according to the scanning order, and obtaining the spectral vector sequence.

[0009] Optionally, the KAN-Transformer fusion coding module is used when performing cross-modal attention interaction on the spatial features of the hyperspectral image, the synthetic aperture radar image, or the lidar image; The KAN-Transformer fusion encoding module performs cross-modal attention interaction by: using the spatial features of the synthetic aperture radar image or lidar image as a query, using the final hidden state as a key and value, calculating cross-attention, and outputting the fused features.

[0010] Optionally, the KAN-Transformer fusion coding module includes a KANLinear layer and a Transformer encoder, wherein the KANLinear unit is integrated into the two-layer perceptron of each Transformer encoder.

[0011] Optionally, the KANLinear unit performs an independent nonlinear transformation on each input dimension by replacing the standard fully connected mapping with a learnable univariate spline function.

[0012] Optionally, the KANLinear unit is: ; in, and These are the basic activation function and the univariate spline activation function, respectively. and These are the basic weight matrix and the univariate spline weight matrix, respectively. y is the output of the KANLinear layer, and x is the input feature.

[0013] Optionally, training the SMKT-Net network using a knowledge distillation training strategy includes: The SMKT-Net network is trained using the standard cross-entropy and the KL divergence of the teacher-student probability distribution to obtain the distillation loss.

[0014] The beneficial effects of this invention are as follows: This invention achieves simultaneous modeling of spectral details and spatial structure information within a unified architecture, providing a new solution for multi-source, multi-modal remote sensing classification; This invention designs and implements an efficient spiral scanning Mamba encoding module, which constructs local sequences from the central pixel along a spiral scanning path, ensuring comprehensive neighborhood coverage while maintaining spatial proximity, achieving efficient capture of local textures and long-distance spatial dependencies, and enhancing the accuracy of spectral-spatial feature representation; This invention proposes a cross-modal KAN-Transformer nonlinear fusion mechanism, which achieves joint representation of multi-source, multi-modal remote sensing features by performing learnable nonlinear mapping and cross-modal cross-attention fusion on multi-modal spectral and spatial features, ensuring deep alignment between different modal remote sensing features while enhancing the cross-modal feature nonlinear modeling fusion capability and training stability. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a general framework diagram of the SMKT-Net network according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the feature acquisition scanning method according to an embodiment of the present invention, wherein (a) is a traditional linear scan and (b) is a spiral scan; Figure 3 This is a schematic diagram of the feedforward network layer in an embodiment of the present invention, wherein (a) is a traditional MLP and (b) is a KAN nonlinear transform. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] This embodiment proposes a multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation, including: Acquire multi-source, multi-modal remote sensing data, which includes hyperspectral images (HSI), synthetic aperture radar (SAR) images, and lidar (LiDAR) images; The multi-source multimodal remote sensing data is input into the SMKT-Net network. The SMKT-Net network performs helical scanning processing on the hyperspectral image to obtain hyperspectral image features. The hyperspectral image features and the spatial features of the synthetic aperture radar image or lidar image are subjected to cross-modal attention interaction to obtain fused features. The SMKT-Net network is trained through a knowledge distillation training strategy. The fused features are input into a classifier to obtain the probability of the target category.

[0020] Specifically, such as Figure 1 As shown, the SMKT-Net network framework consists of four parts: an HSI spectral-spatial feature encoding branch, a SAR / LiDAR feature extraction branch, a Transformer-based fusion module, and a classification head. First, the spatial order dependencies of HSI hyperspectral modes are captured using helical scanning Mamba. This is supplemented by spatial features obtained from SAR / LiDAR data through channel dimension compression and feature aggregation using adaptive average pooling, which are then used as the model input features. Subsequently, the generated multi-source, multi-modal feature maps are imported into the KAN-Transformer fusion module, which uses a learnable nonlinear function, for cross-modal attention interaction. Finally, the fused feature representation is input into the classifier for pixel-by-pixel target category prediction. A knowledge distillation training strategy is employed to further improve model performance without increasing inference complexity.

[0021] Furthermore, the helical scanning process performed on the hyperspectral image includes: Acquire the target pixel in the hyperspectral image, and extract the local image patch around the target pixel; The local image patch is spiral-scanned to obtain a spectral vector sequence; The spectral vector sequence is processed using a selective state-space model to aggregate spectral-spatial information and obtain the final hidden state, i.e., the hyperspectral image features. The final hidden state contains all the spectral-spatial features of the neighborhood of the central pixel.

[0022] Furthermore, spiral scanning of a local image patch includes: starting from the first pixel and spiraling inwards to the center pixel, arranging the spectral vectors of the pixels into a sequence according to the scanning order, and obtaining the spectral vector sequence.

[0023] Specifically, HSI possesses rich spectral information, and fully utilizing its spatial context information is crucial for accurate remote sensing scene classification. To this end, a spiral scanning Mamba module is proposed, which focuses on different class centers and encodes the neighborhood information of each pixel in the HSI. Compared with traditional linear scanning methods (… Figure 2 Compared to (a), spiral scanning has higher efficiency in acquiring contextual information. First, for a given target pixel, a local image patch of size P×P (containing all spectral bands within that region) is extracted. Then, a spiral scan is performed on this image patch. For example... Figure 2 As shown in (b), the spectral vectors of the pixels are arranged in sequence according to the scanning order, spiraling inward from the first pixel to the center pixel. This scanning method ensures full coverage of the local area and preserves the spatial proximity relationship between pixels in the sequence, which can more efficiently capture the spatial context information required for remote sensing scene classification.

[0024] Assuming the spectral vector sequence obtained by this helical sequence is , The helical Mamba module uses a selective state-space model (SSM) (Gu and Dao, 2023) to process the above sequence in order to aggregate spectral-spatial information. At each time step of the sequence... SSM is based on the previous hidden state and current input Update hidden status : (1); In the formula To define the learnable state matrix of a linear recursive relation, For the first The intermediate output of the step. The initial hidden state is set to... After processing the entire sequence, the final hidden state is... This is the final representation output of the HSI mode for this image patch. Hidden State Accumulated from step 1 to step 2 The context information of all pixels in the step sequence, i.e., the compressed representation of the current spiral sequence. Therefore, after completely scanning the image patch, the final hidden state... It encapsulates all the spectral-spatial features of the central pixel's neighborhood, which can be output as the positional feature vector of that pixel in the HSI spectral-spatial feature encoding branch. Therefore, the spiral scanning Mamba sequence modeling ensures that the network effectively captures local textures and long-range spatial dependencies in the HSI modality.

[0025] Furthermore, when performing cross-modal attention interaction on the spatial features of the hyperspectral image, the synthetic aperture radar image, or the lidar image, the KAN-Transformer fusion coding module is used. The KAN-Transformer fusion encoding module performs cross-modal attention interaction by: using the spatial features of the synthetic aperture radar image or lidar image as a query, using the final hidden state as a key and value, calculating cross-attention, and outputting the fused features.

[0026] Furthermore, the KAN-Transformer fusion coding module includes a KANLinear layer and a Transformer encoder, wherein the KANLinear unit is integrated into the two-layer perceptron of each Transformer encoder.

[0027] Furthermore, the KANLinear unit performs an independent nonlinear transformation on each input dimension by replacing the standard fully connected mapping with a learnable univariate spline function.

[0028] Specifically, to fully integrate multi-source, multi-modal features from HSI and SAR / LiDAR, a KAN-Transformer cross-modal dynamic fusion coding module was designed. This module takes feature tokens from both modalities as input: features from the HSI branch (output of the spiral Mamba module)... The system also includes SAR / LiDAR features at the corresponding locations. Furthermore, a cross-modal Transformer encoder is employed to extend the self-attention mechanism to cross-attention between HSI and SAR / LiDAR features.

[0029] Specifically, in the cross-attention layer, SAR / LiDAR features are treated as queries, and HSI features are treated as keys and values, respectively, thus achieving bidirectional feature interaction. Let the query matrix of the SAR / LiDAR branch be... The key and value matrices of the HSI branch are respectively and Then the output of cross-attention is: (2); In the formula The feature dimension is defined by cross-attention computation, which enables the network to effectively capture complementary information across modalities, thereby obtaining a joint representation that integrates spectral and elevation features.

[0030] like Figure 3 As shown in (a)-(b), in the feedforward network of the Transformer (such as the MLP), the standard fully connected mapping is used... Instead, learnable univariate spline functions are used to achieve independent nonlinear transformations for each input dimension. Compared to the fixed activation function (linear weights) of traditional MLPs, this not only provides greater flexibility in function approximation but also allows for modeling complex nonlinear relationships with fewer parameters, improving model stability. The KANLinear layer can be represented as: (3); In the formula, y represents the output of the KANLinear layer. and These are the basic activation function and the univariate spline activation function, respectively. and These are the basic weight matrix and the univariate spline weight matrix, respectively. Features can be defined by inputting learnable B-spline functions: (4); In the formula Indicates input features The calculated B-spline basis functions are as follows: (5); In the formula These are recursively defined B-spline basis functions. The nodes represent the spline. It is the order of the spline. For the spline coefficients, the updated B-spline basis functions ensure that the model can efficiently capture richer nonlinear feature relationships in multi-source, multimodal remote sensing data.

[0031] Specifically, KANLinear units are integrated into the two-layer perceptron of each Transformer encoder block to construct a KAN-enhanced Transformer layer (KAN-Transformer). The entire fusion module stacks several such KAN-Transformer encoding layers. The final encoder output is a unified set of fused features that contain both spectral and SAR / LiDAR information. These fused features are then input into the classification head to predict the probability of the output target class.

[0032] Furthermore, training the SMKT-Net network using the knowledge distillation training strategy includes: The SMKT-Net network is trained using the standard cross-entropy and the KL divergence of the teacher-student probability distribution to obtain the distillation loss.

[0033] Specifically, during model training, a teacher-student knowledge distillation strategy based on joint supervision using standard cross-entropy-probability distribution KL divergence loss is employed to optimize the model. Both the teacher and student models utilize the same SMKT-Net network backbone, including spiral Mamba and KAN-Transformer modules. Specifically, both the real labels and the soft output of the teacher model are used as supervisory signals for training the teacher and student networks.

[0034] Assuming the teacher model is subjected to temperature The smoothed output class probability distribution is The predicted distribution of the student model is Then the distillation loss can be defined using the standard cross-entropy and the KL divergence of the teacher-student probability distribution: (6); In the formula Predicting for students With real labels Cross-entropy between Distribution of teachers With student distribution KL divergence between them The hyperparameters used to balance the two losses are... , This is the distillation temperature parameter used to smooth the output distribution.

[0035] By using knowledge distillation based on joint supervised loss, the generalization behavior of the constructed cross-modal SMTD-Net model (teacher) for multi-source multimodal remote sensing scene classification can be transferred to the distillation model (student), thereby ensuring that the distillation model can maintain a compact size while achieving high classification accuracy.

[0036] This embodiment conducted extensive experiments on four publicly available benchmark multimodal remote sensing fusion datasets (Houston, Trento, MUUFL, and Changbai Mountain volcano dataset), verifying the excellent classification performance and applicability of the proposed cross-modal SMKT-Net framework for homogeneous and topographically complex multimodal remote sensing images in terms of both quantitative and qualitative aspects (OA, mF1, mIoU).

[0037] Example 1: This embodiment applies the method proposed in this invention to a specific scenario of multi-source, multi-modal remote sensing lithology classification in the Changbai Mountain region. The Changbai Mountain volcanic area has significant topographic relief, complex vegetation cover, and different lithologies (such as basalt and trachyte) exhibit high heterogeneity in spectral and spatial structure. This embodiment achieves high-precision lithology classification through the following steps or modules: 1. Acquisition and preprocessing of multi-source lithological data: It was used to acquire hyperspectral imagery (HSI) and synthetic aperture radar (SAR) data for the Changbai Mountain study area. HSI provides fine spectral fingerprint information of rock mineral composition (such as spectral differences in different oxide contents); SAR data provides elevation and texture information of landforms such as craters and lava plateaus.

[0038] The acquired data was subjected to radiometric correction, geometric correction, and registration, and then cropped into local image patches containing the central pixel and its neighborhood (11). (11-size), construct a multi-source lithological sample set.

[0039] 2. SMKT-Net lithological feature extraction and fusion: The SMKT-Net network is constructed and used to perform feature modeling on the above multi-source data, specifically including: HSI Spectral-Spatial Feature Encoding: To address the spatial continuity of rock distribution in Changbai Mountain, a spiral scanning Mamba module is used to perform spiral serialization processing on hyperspectral image blocks from the outside in. This process preserves the spatial distribution structure of lithology while using a selective state space model (SSM) to capture the spectral variation characteristics of rocks over long distances, thus solving the problem of different spectra of the same material in complex terrain.

[0040] Specifically, the helical scanning process for the hyperspectral image includes: Acquire the target pixel in the hyperspectral image, and extract the local image patch around the target pixel; The local image patch is spiral-scanned to obtain a spectral vector sequence; The spectral vector sequence is processed using a selective state-space model to aggregate spectral-spatial information and obtain the final hidden state, i.e., the hyperspectral image features. The final hidden state contains all the spectral-spatial features of the neighborhood of the central pixel.

[0041] SAR spatial structure feature extraction: Based on the rock surface roughness and terrain elevation information reflected by radar data, adaptive average pooling is used to perform channel compression and feature aggregation to extract robust spatial structure features as auxiliary information.

[0042] Cross-modal nonlinear fusion: Utilizing the KAN-Transformer fusion encoding module, the spatial structure features of SAR are used as the query, and the spectral-spatial features of HSI are used as the key and value. Nonlinear transformation is performed through the learnable spline function of the KANLinear layer, enhancing the model's ability to fit nonlinear lithological boundaries and achieving deep interaction between spectral and shape features.

[0043] 3. Knowledge distillation optimization and classification: Training Strategy: A knowledge distillation strategy is employed, using a complex SMKT-Net trained on large-scale lithological data as the teacher network to guide a lightweight student network in learning the distribution patterns of lithology in Changbai Mountain. KL divergence loss is utilized to enable the student network to mimic the probability distribution of the teacher network, thereby improving inference speed in practical field applications while maintaining classification accuracy.

[0044] Classification output: The fused features are input into the classifier, which outputs the probability of the lithology category corresponding to each pixel (such as alkaline flow rock, trachyte, basalt, etc.), and finally generates a fine lithology classification map of the Changbai Mountain study area.

[0045] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation, characterized in that, include: Acquire multi-source, multi-modal remote sensing data, wherein the multi-source, multi-modal remote sensing data includes hyperspectral images, synthetic aperture radar images, and lidar images; The multi-source multimodal remote sensing data is input into the SMKT-Net network. The SMKT-Net network performs helical scanning processing on the hyperspectral image to obtain hyperspectral image features. The hyperspectral image features and the spatial features of the synthetic aperture radar image or lidar image are subjected to cross-modal attention interaction to obtain fused features. The SMKT-Net network is trained through a knowledge distillation training strategy. The fused features are input into a classifier to obtain the probability of the target category.

2. The multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation according to claim 1, characterized in that, Helical scanning processing of the hyperspectral image includes: Acquire the target pixel in the hyperspectral image, and extract the local image patch around the target pixel; The local image patch is spiral-scanned to obtain a spectral vector sequence; The spectral vector sequence is processed using a selective state-space model to aggregate spectral-spatial information and obtain the final hidden state, i.e., the hyperspectral image features. The final hidden state contains all the spectral-spatial features of the neighborhood of the central pixel.

3. The multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation according to claim 2, characterized in that, The spiral scanning of the local image block includes: scanning the local image block spirally inward from the first pixel to the center pixel, arranging the spectral vectors of the pixels into a sequence according to the scanning order, and obtaining the spectral vector sequence.

4. The multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation according to claim 1, characterized in that, When performing cross-modal attention interaction on the spatial features of the hyperspectral image, the synthetic aperture radar image, or the lidar image, the KAN-Transformer fusion coding module is used. The KAN-Transformer fusion encoding module performs cross-modal attention interaction by: using the spatial features of the synthetic aperture radar image or lidar image as a query, using the final hidden state as a key and value, calculating cross-attention, and outputting the fused features.

5. The multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation according to claim 1, characterized in that, The KAN-Transformer fusion coding module includes a KANLinear layer and a Transformer encoder, wherein the KANLinear unit is integrated into the two-layer perceptron of each Transformer encoder.

6. The multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation according to claim 5, characterized in that, The KANLinear unit performs an independent nonlinear transformation on each input dimension by replacing the standard fully connected mapping with a learnable univariate spline function.

7. The multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation according to claim 5, characterized in that, The KANLinear unit is: ； in, and These are the basic activation function and the univariate spline activation function, respectively. and These are the basic weight matrix and the univariate spline weight matrix, respectively. y is the output of the KANLinear layer, and x is the input feature.

8. The multi-source, multi-modal remote sensing image classification method based on Mamba spiral scanning and knowledge distillation according to claim 1, characterized in that, Training the SMKT-Net network using a knowledge distillation training strategy includes: The SMKT-Net network is trained using the standard cross-entropy and the KL divergence of the teacher-student probability distribution to obtain the distillation loss.

Images