Multi-source Remote Sensing Image Classification Method Based on Spectrum Adaptive Feature Fusion

Abstract

The present invention relates to a multi-source remote sensing image classification method based on spectrum adaptive feature fusion, belonging to the field of remote sensing image processing and analysis. Aiming at the problems existing in the existing multi-source remote sensing image classification methods, such as poor flexibility in feature extraction, suboptimal feature fusion strategies, and low computational efficiency, the present invention first preprocesses and divides the multi-source remote sensing data set, then constructs a spectrum adaptive feature fusion neural network model, then conducts model training and quantitative evaluation, and finally analyzes the visualization results of the model. Through the effective extraction and fusion of the features of hyperspectral images and lidar / synthetic aperture radar images, the classification of multi-source remote sensing data is realized, which is applicable to multi-source data fusion and classification scenarios, can meet the remote sensing data processing requirements of complex surface information, has significant advantages in improving classification accuracy and processing efficiency, and provides a more efficient and accurate solution for multi-source remote sensing image classification.

Description

Technical Field

[0001] This technology belongs to the field of remote sensing image processing and analysis, and particularly relates to a multi-source remote sensing image classification method based on spectrum adaptive feature fusion. Background Art

[0002] Multi-source remote sensing image classification refers to the classification and recognition of surface information using various types of remote sensing data (such as hyperspectral images (HSIs), light detection and ranging (LiDAR), and synthetic aperture radar (SAR), etc.). This technology is widely applied in fields such as land use and cover classification, agricultural monitoring, forest and ecological environment monitoring, urban planning, disaster monitoring, and water resource management. Hyperspectral images can identify subtle differences between different substances due to their rich spectral information; LiDAR technology can accurately measure the three-dimensional structure of terrain and ground objects, with high spatial resolution and accuracy; while SAR has the ability to image under all-weather and various weather conditions and can penetrate clouds and vegetation. However, each data type also has its own deficiencies. Hyperspectral imaging is sensitive to atmospheric and lighting conditions, and cloud cover or insufficient lighting will affect data quality; LiDAR mainly provides height and structure information and lacks spectral information, making it difficult to distinguish substances with similar structures but different spectral characteristics; SAR is sensitive to the geometric shape and orientation of objects, and the same object may exhibit different radar echo characteristics in different orientations, thus increasing the classification difficulty. Therefore, comprehensively utilizing the advantages of multi-source remote sensing images helps to improve the accuracy and robustness of classification tasks.

[0003] In the field of multi-source remote sensing image classification, the joint analysis and processing of different data sources are crucial for improving classification accuracy. However, due to the significant differences in data dimensions and feature representations among different data sources (such as hyperspectral images, LiDAR, SAR, etc.), how to effectively fuse these heterogeneous data has become the core challenge in current research. In recent years, the rapid development of deep learning technologies has provided new solutions for the fusion and classification of multi-source data. In particular, convolutional neural networks (CNNs) and Transformers, as two major mainstream models, have demonstrated powerful processing capabilities in multi-source remote sensing image classification tasks. Specifically, the CNN method performs outstandingly in processing classification tasks of different data sources. With its excellent local spatial feature extraction ability, CNN can extract and fuse the spatial and spectral features of multi-source data layer by layer, thus significantly improving classification accuracy. On the other hand, Transformers can capture data associations within the global scope through the self-attention mechanism, and are particularly suitable for processing high-dimensional and multi-source data such as hyperspectral images. Transformers are good at global modeling and can more comprehensively integrate the feature information of different data sources to improve classification performance. Generally speaking, CNN focuses more on the extraction of local features, while Transformers achieve fine-grained feature fusion through global correlation modeling. Both have their own advantages in multi-source remote sensing image classification tasks.

[0004] In the field of multi-source remote sensing image classification, although significant progress has been made in deep learning technologies in recent years, there are still some unsolved problems, which are summarized as follows: (1) Insufficient flexibility in feature extraction: Existing feature extraction methods have limitations in adapting to the frequency features of different types of input data, resulting in the inability to effectively capture the subtle changes in the data. (2) Poor feature fusion strategy: Existing methods fail to fully utilize the advantages of each modality such as the spectral features of hyperspectral images (HSIs) and the spatial features of LiDAR / SAR during the feature fusion process, resulting in unsatisfactory fusion effects between different modalities. (3) Poor computational efficiency: Although the Transformer model performs excellently in capturing global features, its high computational complexity and significant computational cost limit its widespread adoption in practical applications.

[0005] In view of this, there is an urgent need to develop more flexible and efficient feature extraction and fusion methods to improve the overall performance of multi-source remote sensing image classification. Summary of the Invention

[0006] Aiming at the problems of insufficient flexibility in feature extraction, poor feature fusion strategy, and poor computational efficiency existing in the existing multi-source remote sensing image classification methods, the present invention proposes a multi-source remote sensing image classification method based on spectral adaptive feature fusion to improve the accuracy and performance of multi-source remote sensing image classification.

[0007] To achieve the above objectives, the technical solutions implemented in the present invention are as follows:

[0008] A multi-source remote sensing image classification method based on spectral adaptive feature fusion, comprising the following steps:

[0009] S1: Preprocessing of multi-source remote sensing data sets: Hyperspectral image dimensionality compression, data filling; After preprocessing, it is divided into a training set and a test set;

[0010] S2: Construct a spectral adaptive feature fusion neural network model, which includes a dynamic frequency domain feature extraction module (Dynamic Frequency Domain Feature Extraction Module), a spectral-spatial adaptive feature fusion module (Spectral-Spatial Adaptive Feature Fusion Module), and a fully connected layer for classification; The data processing process of this network model is expressed as input data → preliminary feature extraction by convolutional blocks → dynamic frequency domain feature extraction module → spectral-spatial adaptive feature fusion module → fully connected layer; The dynamic frequency domain feature extraction module is designed to extract frequency domain features and enhance the expressive ability of data through a dynamic Fourier transform filtering process; The spectral-spatial adaptive feature fusion module fuses data features from different sources, and the fully connected layer for classification is located at the last stage of the network model and is used to perform classification based on the results of previous feature extraction and fusion;

[0011] S3: Train the spectral adaptive feature fusion neural network model, use the cross-entropy loss function, set a weight decay term in the optimizer, and constrain the complexity of model parameters by applying L2 regularization to the weights;

[0012] S4: Use the trained spectral adaptive feature fusion neural network model to process multi-source remote sensing data and perform visual result analysis.

[0013] Further, the specific content of S1 includes:

[0014] S1-1: Read the hyperspectral image HSI and LiDAR / SAR data, and perform dimensionality reduction processing on the hyperspectral data using the principal component analysis PCA method;

[0015] S1-2: Perform filling operations on the input hyperspectral image, the image after PCA dimensionality reduction, and LiDAR / SAR data;

[0016] S1-3: Use a window of a specified size to slide on the filled data to generate image patches corresponding to each position;

[0017] S1-4: Construct a PyTorch dataset class and generate training and test sets through groundTruth labels and corresponding position indices.

[0018] Furthermore, in S2, the data processing process of the adaptive feature fusion neural network model is as follows:

[0019] S2-1: The input data is HSI and LiDAR / SAR data. The dimension of the input data HSI is (B, C, N, P, P), where B is the batch size, C is the number of channels (usually 1), N is the number of hyperspectral bands, and P is the size of the cropped image patch, i.e., WindowSize. The dimension of the input data of the LiDAR / SAR image is (B, C, P, P), where B is the batch size, C is the number of channels, and P is the size of the cropped image patch;

[0020] S2-2: The dynamic frequency domain feature extraction module is mainly responsible for extracting effective feature information from the input data: for HSI data, the data stream first undergoes preliminary feature extraction: HSI data → 3D Conv → BatchNorm3D → ReLU activation function; similarly, for LiDAR / SAR data, the data stream also undergoes preliminary feature extraction: LiDAR / SAR data → 2DConv → BatchNorm2D → ReLU activation function; the calculation process of the preliminary feature extraction operation is expressed as:

[0021] ;

[0022] ;

[0023] where represents the input data of HSI, represents the LiDAR / SAR data. represents the preliminarily extracted hyperspectral features, represents the preliminarily extracted LiDAR / SAR features;

[0024] S2-3: For the obtained and use the dynamic frequency domain feature extraction module (Dynamic FourierFiltering Block) to extract features adapted to the data modality. The overall structure is similar to the conventional structure of the Transfomer Block. The overall structure is input feature → LayerNorm → dynamic Fourier filtering → residual connection → LayerNorm → MLP → residual connection. The entire calculation process is as follows:

[0025] ;

[0026] ;

[0027] Among them represents LayerNorm, represents the DynamicFilter, and MLP is a multi-layer perceptron. The composition structure is Linear → ReLU → Dropout → Linear → Dropout. The calculation process of MLP is as follows:

[0028] ;

[0029] The processing process of is similar to the processing process of

[0030] S2-3: The dynamic filter is the core module of the dynamic Fourier filtering block. The filter is a learnable parameter , represents the complex domain, represents the number of filters, and represent the length and width of the input data.

[0031] ;

[0032] ;

[0033] ;

[0034] Among them, represents the input data, which is or here. GAP represents global average pooling, and SoftMax is used for data normalization. represents the Fourier transform, is the inverse Fourier transform. is a learnable parameter, that is, the filter, is The dynamic filter kernel generated by processing. This filter kernel is multiplied by the features after the input passes through the linear layer and the Fourier transform to obtain the processed features, and then becomes the real domain through the inverse Fourier transform, and then passes through the linear layer to obtain the final output .

[0035] S2-4: The process of the data passing through the preliminary feature extraction and the dynamic frequency domain feature extraction module will be repeated times. Generally, the value of L ranges from 1 to 2. After the above feature extraction process, and It is sent to the spectral-spatial adaptive feature fusion module for fusion. The fusion process is as follows:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] Among them, CA represents Channel Attention, SA represents Spatital Attention, represents the concatenation operation, is an activation function commonly used in deep learning, is a weight map obtained through learning, which is used to guide the network to make up for the advantages of HSI and LiDAR / SAR data, so as to obtain better fusion features. and represent convolutions with convolution kernel sizes of 5 and 1 respectively.

[0041] The detailed calculation processes of channel attention and spatial attention are as follows:

[0042] ;

[0043] ;

[0044] Among them, GAP is global average pooling, and GMP is global max pooling.

[0045] Furthermore, in the above S3, the model training includes:

[0046] (1) Optimizer settings. The initial learning rate is generally set to 0.0001 to control the step size of each parameter update. The choice of learning rate affects the convergence speed and stability of the model, and usually needs to be tuned according to the specific dataset and task requirements. The default momentum coefficient betas = (0.9, 0.999) is used to control the first-order moment estimate (momentum) and the second-order moment estimate (exponentially weighted average of the gradient square) of the gradient respectively. The momentum parameter helps to suppress gradient oscillation during the optimization process and accelerate convergence. To prevent overfitting, a weight decay term weight_decay is set in the optimizer. This term imposes L2 regularization on the weights to constrain the complexity of the model parameters. The numerical stability parameter eps = 1e-8 is set to avoid the division-by-zero problem in the denominator of the update formula, thereby enhancing the stability of the optimization process.

[0047] (2)Loss function design. To effectively guide the update of the model's parameters, the CrossEntropyLoss function is adopted as the optimization objective. The cross-entropy loss function is a loss function widely used in multi-class classification tasks. Its core idea is to maximize the prediction probability of the model for the correct class by measuring the difference between the predicted distribution and the true distribution. The mathematical expression of the cross-entropy loss function is as follows:

[0048] ;

[0049] where N is the number of samples, C is the number of classes, represents the distribution of the true label of the i-th sample in the j-th class (usually 0 or 1), represents the predicted probability of the i-th sample in the j-th class.

[0050] (3)Forward propagation and loss calculation. The input data includes the dimensionality-reduced hyperspectral image hsi_pca and other feature data Xdata. After passing through the model calculation, the output outputs are generated, representing the predicted probability distribution of the input data over various classes. The loss function is used to measure the difference between the predicted results of the model output and the true labels, providing guidance for model optimization.

[0051] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0052] 1. The dynamic frequency-domain feature extraction module proposed in the present invention dynamically generates global filters through fast Fourier transform, which can be flexibly adjusted according to the frequency distribution of the input features, so as to adapt to the frequency characteristics of different images. This dynamic generation mechanism significantly enhances the flexibility and effectiveness of feature extraction. Compared with the traditional multi-head self-attention mechanism, the present invention uses fast Fourier transform to achieve efficient capture of global features. While retaining the global feature modeling ability of the multi-head self-attention mechanism, it greatly reduces the computational complexity, is particularly suitable for the processing tasks of high-resolution images, and greatly improves the computational efficiency.

[0053] 2. The spectral-spatial adaptive fusion module of the present invention dynamically learns the fusion weights between different data sources (such as hyperspectral images and LiDAR / SAR images) by introducing the attention mechanism. Different from the traditional simple splicing or superposition methods, this module can adaptively extract and fuse the spectral features of hyperspectral images and the spatial features of LiDAR / SAR images according to the characteristics of the data. By dynamically optimizing the fusion weights, the complementarity of different data sources is fully exploited, overcoming the limitations that may exist in a single data source, thus significantly enhancing the adaptability and classification accuracy of the model in complex tasks.

[0054] 3. Through the collaborative design of the dynamic frequency-domain feature extraction module and the spectral-spatial adaptive fusion module, the present invention not only retains the global feature capture ability but also significantly reduces the computational complexity. At the same time, through the adaptive data fusion strategy, the effective integration and feature enhancement of multi-source data are achieved. This design enables the present invention to exhibit more excellent performance when processing multi-source data and significantly reduces the consumption of computing resources.

[0055] The present invention realizes the classification of multi-source remote sensing data by effectively extracting and fusing the features of hyperspectral images (HSIs) and Light Detection and Ranging (LiDAR) / Synthetic Aperture Radar (SAR) images. This method is applicable to the fusion and classification scenarios of multi-source data, can meet the remote sensing data processing requirements of complex and diverse surface information, and has significant advantages in improving classification accuracy and processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic flowchart of an embodiment of the present invention.

[0057] Figure 2 It is a schematic flowchart of the principle of an embodiment of the present invention.

[0058] Figure 3 It is a schematic diagram of the neural network structure described in an embodiment of the present invention.

[0059] Figure 4 It is a detailed structural schematic diagram of the dynamic frequency-domain feature extraction module in an embodiment of the present invention.

[0060] Figure 5 It is a detailed structural schematic diagram of the spectral-spatial adaptive feature fusion module in an embodiment of the present invention.

[0061] Figure 6 It is a schematic diagram of the visualization images of multiple algorithms on the Berlin dataset in an embodiment of the present invention.

[0062] Figure 7 It is a schematic diagram of the visualization images of multiple algorithms on the Houston2013 dataset in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] The present invention will be further explained and illustrated below through specific embodiments in conjunction with the accompanying drawings.

[0064] Embodiment 1:

[0065] A multi-source remote sensing image classification method based on spectral adaptive feature fusion, referring to Figure 1 , includes the following steps:

[0066] (1) Preprocessing and partitioning of the multi-source remote sensing dataset:

[0067] (1.1)The dimensionality reduction of hyperspectral images is the primary step in multi-source remote sensing data processing. Hyperspectral images have extremely high spectral resolution, and each pixel contains multiple spectral channels, which enables them to accurately capture the subtle spectral features of objects. However, when the overly high spectral dimension is directly used for training, it will cause a sharp increase in computational complexity. Since hyperspectral images contain multiple spectral channels, directly using them for training will lead to a huge increase in computational complexity. Therefore, principal component analysis (PCA) is used to reduce the dimensionality of hyperspectral data, compressing the high-dimensional spectral data into fewer principal components (usually set to 30 channels), while retaining most of the key information, significantly reducing the computational burden, and improving the processing efficiency.

[0068] (1.2)To ensure the complete retention of edge information when extracting samples with a sliding window, reflectance padding is performed on the hyperspectral images (original and PCA-dimension-reduced) and LiDAR / SAR data. Reflectance padding effectively extends the data based on edge features, avoids information loss, guarantees data integrity and accuracy, and lays a foundation for generating high-quality samples.

[0069] (1.3)A sliding window of a specified size (usually 11 and odd) slides over the padded data to generate image patch samples corresponding to each position, covering the original hyperspectral data, PCA-dimension-reduced data, and LiDAR / SAR data. This method fully integrates multi-source data, provides rich resources for constructing a comprehensive and accurate dataset, and facilitates the mining of potential features and laws.

[0070] (1.4)Constructing the PyTorch dataset class and generating the training and test sets. The PyTorch dataset class is constructed, and the training and test sets are generated through the groundTruth labels and corresponding position indices. The sample numbers of the used Berlin and Houston2013 datasets are listed in detail in Table 1 and Table 2 respectively. Through this construction and generation process, the data can be effectively organized to meet the requirements of deep learning model training and evaluation, thus providing reliable data support and guarantee for subsequent model training, optimization, and performance evaluation.

[0071] Table 1 Numbers of training and test samples of the Berlin dataset

[0072] 。

[0073] Table 2 Numbers of training and test samples of the Houston2013 dataset

[0074] 。

[0075] (2)Constructing a spectral adaptive feature fusion neural network model

[0076] As shown in Figure 3, the network structure of the present invention consists of a dynamic frequency domain feature extraction module, a spectral-spatial adaptive feature fusion module, and a classification fully connected layer. The dynamic frequency domain feature extraction module enhances the frequency domain feature expression ability of HSI and LiDAR / SAR data through the dynamic filtering process of Fourier transform; the spectral-spatial adaptive feature fusion module utilizes the complementarity of HSI and LiDAR / SAR data to dynamically fuse the features of the two types of data; the classification fully connected layer is located at the end of the network and is responsible for completing the classification task based on the extracted and fused features. The process of the overall network structure is: input data → preliminary feature extraction by convolutional block → dynamic frequency domain feature extraction module → spectral-spatial adaptive fusion module → fully connected layer. The input data includes two parts, namely HSI and LiDAR / SAR, where the input dimension of HSI is (B, C, N, P, P), and the input dimension of LiDAR / SAR is (B, C, P, P). Among them, B represents the batch size, C is the number of channels, N is the number of hyperspectral bands, and P is the size of the cropped image patch (i.e., WindowSize).

[0077] (2.1) The dynamic frequency domain feature extraction module is mainly responsible for extracting effective feature information from the input data. For HSI data, the data stream first undergoes preliminary feature extraction: HSI data → 3D Conv → BatchNorm3D → ReLU activation function. Similarly, for LiDAR / SAR data, the data stream also undergoes preliminary feature extraction: LiDAR / SAR data → 2DConv → BatchNorm2D → ReLU activation function. The calculation process of the preliminary feature extraction operation can be expressed as:

[0078] ;

[0079] ;

[0080] where represents the input data of HSI, represents the LiDAR / SAR data. and represent 3D and 2D batch normalization respectively. represents the preliminarily extracted hyperspectral features, represents the preliminarily extracted LiDAR / SAR features.

[0081] (2.2) As Figure 4 shown, for the extracted and Extract features adapted to their respective data modalities through the Dynamic Fourier Filtering Block. The overall structure of this block is similar to the standard structure of the Transformer Block. The specific process is as follows: Input features → LayerNorm → Dynamic Fourier filtering → Residual connection → LayerNorm → MLP → Residual connection. The calculation process is as follows:

[0082] ;

[0083] ;

[0084] Among them represents layer normalization of the LayerNorm layer, represents the DynamicFilter, and MLP is a multi-layer perceptron. Its composition structure is Linear→ReLU→Dropout→Linear→Dropout. The calculation process of MLP is as follows:

[0085] ;

[0086] The processing process of is similar to the processing process of

[0087] (2.3) The dynamic filter is the core module of the dynamic Fourier filtering block. The filter is a learnable parameter , represents the complex domain, represents the number of filters, and represent the length and width of the input data;

[0088] ;

[0089] ;

[0090] ;

[0091] Among them, represents the input data, which is or . GAP represents global average pooling, and SoftMax is used for data normalization. represents the Fourier transform, is the inverse Fourier transform. is a learnable parameter, that is, the filter, is obtained through Process the generated dynamic filter kernel, which is multiplied by the features after the input passes through the linear layer and the Fourier transform to obtain the processed features. Then, the processed features are transformed back to the real domain through the inverse Fourier transform and further passed through the linear layer to obtain the final output 。

[0092] (2.4)As Figure 3 shown, the process of data passing through the preliminary feature extraction and the dynamic frequency domain feature extraction module will be repeated times, and usually the value of L ranges from 1 to 2. As Figure 5 shown, after the above process of feature extraction, and are input into the spectral spatial adaptive feature fusion module for fusion; the fusion process is as follows:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] where CA represents Channel Attention, SA represents Spatital Attention, represents the concatenation operation, is an activation function commonly used in deep learning, is the weight map obtained through learning, which is used to guide the network to make complementary advantages of HSI and LiDAR / SAR data to obtain better fusion features. and represent convolutions with convolution kernel sizes of 5 and 1 respectively.

[0098] The detailed calculation processes of channel attention and spatial attention are as follows:

[0099] ;

[0100] ;

[0101] where GAP is global average pooling and GMP is global max pooling.

[0102] (3)Model Training and Quantitative Evaluation

[0103] (3.1)Model Training Process

[0104] (3.1.1)Optimizer settings. The initial learning rate is usually set to 0.0001, which is used to control the step size of each parameter update. The choice of the learning rate directly affects the convergence speed and stability of the model and needs to be adjusted according to the specific dataset and task requirements. The optimizer uses the default momentum coefficients betas=(0.9,0.999), which control the first-order moment estimate (momentum) and the second-order moment estimate (exponentially weighted average of the gradient squared) of the gradient respectively. The momentum parameter can effectively suppress gradient oscillations and improve the convergence efficiency of optimization. To prevent model overfitting, a weight decay term weight_decay is introduced in the optimizer, which imposes constraints on the weights through L2 regularization to reduce the complexity of the model parameters. At the same time, the numerical stability parameter eps=1e−8 is set to avoid division by zero in the denominator of the update formula, thereby enhancing the stability and robustness of the optimization process.

[0105] (3.1.2)Loss function design. To effectively guide the update of model parameters, the present invention uses the cross-entropy loss function (CrossEntropyLoss) as the optimization objective. Cross-entropy loss is a commonly used loss function for multi-class classification tasks. Its core lies in optimizing the prediction probability of the correct class by measuring the difference between the predicted distribution and the true distribution. Its mathematical expression is as follows:

[0106] ;

[0107] where N is the number of samples, C is the number of classes, represents the distribution of the true label of the i-th sample in the j-th class (usually 0 or 1), represents the predicted probability of the i-th sample in the j-th class.

[0108] (3.1.3)Forward propagation and loss calculation. The input data includes the dimensionality-reduced hyperspectral image hsi_pca and other feature data Xdata. After being processed by the model, these data generate the output outputs, that is, the predicted probability distribution corresponding to each class. Subsequently, the loss function is used to calculate the difference between the model prediction result and the true label, providing guidance for subsequent optimization of model parameters.

[0109] (3.2)Model quantitative evaluation

[0110] (3.2.1)Evaluation process. To comprehensively evaluate the performance of the model in the classification task, the present invention designs a systematic evaluation process. This process is carried out based on the validation dataset after each training Epoch, covering model prediction, performance metric calculation, and result recording, ensuring the scientificity and effectiveness of model optimization.

[0111] (3.2.2)Evaluation metrics

[0112] Accuracy is a metric that measures the proportion of correctly classified samples in each category and is mainly used to evaluate the classification accuracy of a model. For a specific category, accuracy focuses on the proportion of samples that are actually in that category among those predicted by the model to be in that category. A high accuracy indicates that the model has strong discrimination ability in that category.

[0113] Recall measures the ability of the model to successfully identify true samples in each category. It reflects the coverage of the model for samples in a certain category, especially suitable for situations where the category distribution is uneven. A high recall indicates a low false negative rate of the model in a specific category.

[0114] Overall accuracy is the core metric for evaluating the global classification ability of the model, representing the proportion of correctly classified samples among all test samples. It comprehensively considers the performance of all categories and provides a direct and comprehensive measurement method.

[0115] Average precision is the arithmetic mean of the classification recall rates of all categories and is used to measure whether the performance of the model on various categories is balanced. Different from overall accuracy, average precision pays more attention to the performance of the model on minority classes.

[0116] The Kappa coefficient measures the consistency of the classification results by comparing the model's classification results with random classification results. The Kappa coefficient not only considers the classification accuracy but also adjusts for the correct classifications that may be caused by random guessing, so it can more objectively evaluate the classification reliability of the model.

[0117] Class-level metrics (accuracy, recall): These metrics can reveal the performance of the model on specific categories, help discover performance differences between categories, and are suitable for fine-grained category analysis. For example, in the garbage classification task, the ability of the model to identify different types of garbage can be evaluated.

[0118] Global metrics (OA, AA, Kappa coefficient): These metrics provide an overall perspective and reflect the adaptability of the model to the entire dataset. They can be used for global performance comparison and verification of the effect of model improvement.

[0119] Accuracy: Accuracy measures the proportion of correctly classified samples in each category and reflects the recognition accuracy of the model in that category. The calculation formula is as follows:

[0120] ;

[0121] where : The number of correctly classified samples in the i-th category. : The number of samples misclassified as the i-th category.

[0122] Recall: Recall measures the proportion of samples successfully identified in each category, reflecting the coverage ability of the model for that category. The calculation formula is as follows:

[0123] ;

[0124] where : The number of samples in the i-th category that are not correctly classified.

[0125] Overall Accuracy (OA): Overall Accuracy measures the classification accuracy of the model for all test samples, reflecting the overall classification ability of the model. The calculation formula is as follows:

[0126] ;

[0127] where C: The total number of categories. N: The total number of test samples.

[0128] Average Accuracy (AA): Average Accuracy is the arithmetic mean of the recall rates of each category, reflecting the average performance of the model across all categories. The calculation formula is as follows:

[0129] Kappa coefficient.

[0130] Kappa coefficient: The Kappa coefficient measures the degree of agreement between the classification result and random classification, and is used to evaluate the reliability of the classification result. The calculation formula is as follows:

[0131] ;

[0132] where is OA, is the expected value of agreement for random classification, and the calculation formula is:

[0133] ;

[0134] Precision and recall focus on category-level evaluation and can identify specific advantages or deficiencies of the model for certain categories. OA, AA, and the Kappa coefficient provide a global perspective, reflecting the classification performance of the model on the entire dataset.

[0135] (3.2.3) Index Recording and Optimal Preservation

[0136] After each round of training, the overall accuracy (OA) is calculated based on the validation dataset and recorded for subsequent performance comparison and model saving. Subsequently, the OA value of the current validation set is compared with the previously saved historical optimal OA value: if the current OA value exceeds the historical optimal value, the optimal OA is updated and the current model is saved; otherwise, the historical optimal value remains unchanged.

[0137] (3.2.4) Termination Conditions

[0138] If the optimal OA value of the validation set has not improved for 20 consecutive epochs, it is considered that the model performance has stabilized. In this case, the training process will automatically terminate to avoid unnecessary computational overhead and improve training efficiency.

[0139] (4) Analysis of Model Visualization Results

[0140] In this study, the prediction results of the classification model were visualized to generate intuitive color classification images for analyzing the classification effect and spatial distribution characteristics of the model. Input data processing: The model was used to infer the input data (such as hyperspectral images and LiDAR / SAR data) to generate the class prediction results for each pixel. Result mapping: According to the class color mapping rules of different datasets (such as Berlin_color_map, Houston2013_color_map, etc.), the predicted classes were mapped to the corresponding color values to generate color images. Image generation and saving: By updating the image matrix pixel by pixel, the generated color image was finally saved as a.png file using OpenCV, with the file name named after the dataset name and the current timestamp to ensure the uniqueness of the file. The distribution of the classification results was directly shown through the color image, providing an intuitive basis for model performance evaluation.

[0141] Example 2:

[0142] This example further illustrates the effect of the present invention through a simulation experiment:

[0143] The simulation experiment of this embodiment was carried out under the hardware environment of Intel(R) Core(TM) i9-10920X CPU, NVIDIA GeForce RTX 4090, and 32GB of memory, and the software environment of Ubuntu 18.04.6 LTS, Python 3.10.13, and Torch 2.1.1+cu118. The simulation experiment data of the present invention are the Berlin dataset and the Houston2013 dataset. As part of the 2013 IEEE Geoscience and Remote Sensing Society Data Fusion Contest, the Houston2013 dataset provides a unique perspective to observe the urban land cover of Houston, Texas, and its surrounding areas. The dataset records hyperspectral images of 144 bands covering the spectral range of 380 to 1050 nm at a spatial resolution of 2.5 meters, as well as LiDAR data providing precise elevation information, covering 15 different land cover types. The Berlin dataset covers the urban and rural areas of Berlin. The hyperspectral data are the Environmental Mapping and Analysis Program (EnMAP) data simulated based on the hyperspectral imager (HyMap) data. The synthetic aperture radar (SAR) data are the Sentinel-1 dual-polarization (VV-VH) single-look complex (SLC) products obtained from the European Space Agency (ESA). The hyperspectral image (HSI) has 797×220 pixels and 244 spectral bands in the wavelength range of 400 - 2500 nm (reference [berlin]). The synthetic aperture radar image has 1723×476 pixels. The nearest neighbor interpolation method is used to match the spatial resolution of the hyperspectral image and the synthetic aperture radar data.

[0144] The method of the present invention was compared with existing relatively advanced multi-source remote sensing image classification methods. The TBCNN method used in the comparative experiment was proposed in the article "Multisource remote sensing data classification based on convolutional neural network"; the FusAtNet method was proposed in the article "Fusatnet: Dual-attention based spectrospatial multimodal fusion network for hyperspectral and lidar classification". The method was proposed in the article "S2enet: Spatial–spectral cross-modal enhancement network for classification of hyperspectral and lidar data"; the DFINet method was proposed in the article "Hyperspectral and multispectral classification for coastal wetland using depthwise feature interaction network"; the ExViT method was proposed in the article "Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework"; the HCT method was proposed in the article "Joint classification of hyperspectral and LiDAR data using a hierarchical CNN and Transformer".

[0145] It can be seen from Figure 6 that in the processing of the Berlin dataset, the classification map generated by the method adopted in the present invention is highly consistent with the actual situation. Compared with the frequent classification errors of other methods in the industrial area and the area with low vegetation distribution, the method of the present invention shows excellent accuracy, can accurately classify these complex and challenging areas, and is particularly outstanding in retaining fine textures and detailed features, thus making the classification results more accurate and reliable. From Figure 7 it can be seen that when applied to the Houston 2013 dataset, the method of the present invention achieved a more accurate segmentation effect, effectively reduced noise interference in most areas, significantly improved the clarity and accuracy of the overall result, and provided a better data basis and decision-making basis for the analysis and application of this dataset.

[0146] Tables 3 and 4 respectively present the comparison of the method of the present invention with other methods in terms of various indicators under the Berlin dataset and the Houston 2013 dataset. It can be clearly seen from this that the method of the present invention has reached the optimal level in the three key indicators of overall accuracy (OA), average accuracy (AA), and Kappa coefficient. Specifically, in the application of the Berlin dataset, for the classification of specific land cover types such as industrial areas, low vegetation areas, and water areas, the method of the present invention has shown better effects compared to other methods, and can more accurately identify and divide these areas, thus significantly improving the accuracy and reliability of classification. In terms of the Houston 2013 dataset, the classification performance of the method of the present invention in categories such as healthy grassland, stressed grassland, commercial area, highway, and parking lot 2 is also better than other methods, effectively reducing the probability of misclassification and missed classification, providing strong support for the accurate classification and in-depth analysis of this dataset, and further highlighting the advantages and value of the method of the present invention in the classification task of complex datasets.

[0147] Comparison results of performance indicators of each algorithm on the Berlin dataset

[0148] 。

[0149] Comparison results of performance indicators of each algorithm on the Houston 2013 dataset

[0150] 。

[0151] The multi-source remote sensing image classification method based on spectral adaptive feature fusion provided by the present invention is mainly to accurately identify the complex land cover types covered by the multi-source remote sensing image dataset. By adaptively extracting and fusing the spectral features of different source remote sensing data, it effectively improves the accuracy and reliability of image classification, so as to meet the needs of high-precision professional fields such as urban planning, land use monitoring, and environmental resource assessment. However, it is also applicable to other multi-source remote sensing image classification scenarios with similar characteristics, such as the fine classification of agricultural production areas to assist precision agriculture management, the ecological system monitoring of nature reserves to achieve biodiversity protection, and the rapid assessment of disaster areas to support emergency rescue decision-making, etc. In these different application scenarios, the beneficial effects obtained are also similar.

[0152] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as it does not depart from the technical solution content of the present invention, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A multi-source remote sensing image classification method based on spectrum adaptive feature fusion, characterized in that The method includes the following steps: S1: Preprocessing of multi-source remote sensing data sets: dimensionality compression and data filling of hyperspectral images; after preprocessing, it is divided into a training set and a test set; S2: Construct a spectral adaptive feature fusion neural network model, which includes a dynamic frequency domain feature extraction module, a spectral-spatial adaptive feature fusion module, and a fully connected layer for classification; the data processing process of this network model is expressed as input data → preliminary feature extraction by a convolutional block → dynamic frequency domain feature extraction module → spectral-spatial adaptive feature fusion module → fully connected layer; the dynamic frequency domain feature extraction module is designed to extract frequency domain features and enhance the data expression ability through a dynamic Fourier transform filtering process; the spectral-spatial adaptive feature fusion module fuses data features from different sources, and the fully connected layer for classification is located at the last stage of the network model and is used to perform classification according to the results of the previous feature extraction and fusion; in S2, the data processing process of the adaptive feature fusion neural network model is as follows: S2-1: The dimension of the input data HSI is (B, C, N, P, P), where B is the batch size, C is the number of channels, N is the number of hyperspectral bands, and P is the size of the cropped image patch, that is, WindowSize; the dimension of the input data of the LiDAR / SAR image is (B, C, P, P), where B is the batch size, C is the number of channels, and P is the size of the cropped image patch; S2-2: For HSI data, the data stream first undergoes preliminary feature extraction: HSI data → 3DConv → BatchNorm3D → ReLU activation function; for LiDAR / SAR data, the data stream undergoes preliminary feature extraction: LiDAR / SAR data → 2DConv → BatchNorm2D → ReLU activation function; the calculation process of the preliminary feature extraction operation is expressed as: F H = ReLU(BN 3D (Conv 3D (H))) F X = ReLU(BN 2D (Conv 2D (X))) Where H represents the input data of HSI, and X represents LiDAR / SAR data; F H represents the initially extracted hyperspectral features, and F X represents the initially extracted LiDAR / SAR features; S2-3: For the obtained F H and F X Use the dynamic frequency domain feature extraction module to extract features adapted to the data modality. The overall structure is input feature → LayerNorm → dynamic Fourier filtering → residual connection → LayerNorm → MLP → residual connection; the entire calculation process is as follows: F′ H = DF(LN(F H )) + F H F″ H = MLP(LN(F′ H )) + F′ H where LN represents LayerNorm, DF represents the dynamic filter, MLP is a multi-layer perceptron, and its composition structure is Linear → ReLU → Dropout → Linear → Dropout; the calculation process of the MLP is as follows: Y = Drop(Linear(Linear(ReLU(Linear(X))))) F X The processing procedure of H is similar to that of F; S2-4: The filter is a learnable parameter K ∈ C N×H×W , where C represents the complex number field, N represents the number of filters, and H and W represent the length and width of the input data; Routing = SoftMax(MLP(GAP(X))) Among them, X represents the input data, which is F here H or F X , GAP represents global average pooling, and SoftMax is used for data normalization; F represents the Fourier transform, and F -1 is the inverse Fourier transform, K is a learnable parameter, M is a dynamic filter kernel generated through Routing processing, and this filter kernel is multiplied by the features after the input passes through the linear layer and the Fourier transform to obtain the processed features, which are then transformed into the real number domain through the inverse Fourier transform, and then pass through the linear layer to obtain the final output Y; S2-5: The process of the data being processed by the preliminary feature extraction and dynamic frequency domain feature extraction module will be repeated L times, and F H ″ and F X ″ are sent to the spectral space adaptive feature fusion module for fusion, and the fusion process is as follows: F″′ H = CA(F″ H )·F″ H F triple prime X = SA(F double prime X ) · F double prime X W = Sigmoid(Conv 5×5 (Concat(F H ″, F X ″))) F fuse = Conv 1×1 (F″ H + F″ X + W·F″ H +(1 - W)·F″ X ) Among them, CA represents channel attention, SA represents spatial attention, Concat represents the concatenation operation, Sigmoid is a commonly used activation function in deep learning, W is the weight map obtained through learning, and Conv 5×5 and Conv 1×1 represent convolutions with convolution kernel sizes of 5 and 1 respectively; the detailed calculation processes of channel attention and spatial attention are as follows: Y = Linear(ReLU(Linear(GAP(X)))) Y = Conv 5×5 (Concat(GAP(X), GMP(X))) where GAP is global average pooling and GMP is global max pooling; S3: The spectral adaptive feature fusion neural network model is trained, using the cross-entropy loss function, and a weight decay term is set in the optimizer. By applying L2 regularization to the weights, the complexity of the model parameters is constrained; S4: Use the trained spectral adaptive feature fusion neural network model to process multi-source remote sensing data and perform visual result analysis.

2. The multi-source remote sensing image classification method according to claim 1, wherein Specifically, S1 includes: S1-1: Read the hyperspectral image HSI and LiDAR / SAR data, and perform dimensionality reduction processing on the hyperspectral data using the principal component analysis (PCA) method; S1-2: Perform a filling operation on the input hyperspectral image, the image after PCA dimensionality reduction, and the LiDAR / SAR data; S1-3: Slide a window of a specified size over the data that has been filled to generate image patches corresponding to each position; S1-4: Construct a PyTorch dataset class and generate training and test sets through groundTruth labels and corresponding position indices.

3. The multi-source remote sensing image classification method according to claim 1, characterized in that In S3, the model training includes: (1) Optimizer settings: Adjust the learning rate according to the specific dataset and task requirements, and use the default momentum coefficient to control the first-order and second-order moment estimates of the gradient respectively; To prevent overfitting, a weight decay term is set in the optimizer, and by imposing L2 regularization on the weights, the complexity of the model parameters is constrained; (2) Use the cross-entropy loss function as the optimization objective, and the mathematical expression is as follows: where N is the number of samples, C is the number of classes, and y ij represents the distribution of the true label of the i-th sample in the j-th class, and represents the predicted probability of the i-th sample in the j-th class; (3) Forward propagation and loss calculation; After the input data is calculated by the model, the output outputs are generated, representing the predicted probability distribution of the input data on each category.

Images