Multi-source remote sensing image classification method based on spectrum adaptive feature fusion

Abstract

The invention relates to a multi-source remote sensing image classification method based on spectrum adaptive feature fusion, and belongs to the field of remote sensing image processing and analysis. Aiming at the problems of poor feature extraction flexibility, poor feature fusion strategy, low calculation efficiency and the like of the existing multi-source remote sensing image classification method, the method comprises the following steps: firstly, preprocessing and dividing a multi-source remote sensing data set, then constructing a spectrum adaptive feature fusion neural network model, and then carrying out model training and quantitative evaluation; and finally, carrying out model visualization result analysis. The features of a hyperspectral image and a laser radar / synthetic aperture radar image are effectively extracted and fused, classification of multi-source remote sensing data is achieved, the method is suitable for a multi-source data fusion and classification scene, the remote sensing data processing requirement of complex earth surface information can be met, the advantages in the aspect of improving classification precision and processing efficiency are remarkable, and the method is suitable for popularization and application. And a more efficient and accurate solution is provided for multi-source remote sensing image classification.

Description

Technical Field

[0001] The present technology belongs to the field of remote sensing image processing and analysis, and particularly relates to a multi-source remote sensing image classification method based on spectrum adaptive feature fusion. Background Art

[0002] Multi-source remote sensing image classification refers to the classification and identification of surface information using multiple types of remote sensing data (such as hyperspectral images (HSI), laser radar (LiDAR) and synthetic aperture radar (SAR). This technology is widely used in land use and cover classification, agricultural monitoring, forest and ecological environment monitoring, urban planning, disaster monitoring and water resources management. Hyperspectral images can identify subtle differences between different substances with their rich spectral information; LiDAR technology can accurately measure the three-dimensional structure of terrain and objects with high spatial resolution and accuracy; and SAR has the ability to image in all weather conditions and can penetrate clouds and vegetation. However, various data types also have their own shortcomings. Hyperspectral imaging is sensitive to atmospheric and lighting conditions, and cloud cover or insufficient light will affect data quality; LiDAR mainly provides height and structural information, lacks spectral information, and it is difficult to distinguish materials with similar structures but different spectral characteristics; SAR is sensitive to the geometric shape and orientation of objects, and the same object may present different radar echo characteristics in different orientations, thereby increasing the difficulty of classification. Therefore, comprehensively utilizing the advantages of multi-source remote sensing images can help improve the accuracy and robustness of classification tasks.

[0003] In the field of multi-source remote sensing image classification, joint analysis and processing of different data sources are crucial to improving classification accuracy. However, due to the significant differences in data dimensions and feature expressions of different data sources (such as hyperspectral images, LiDAR, SAR, etc.), how to effectively fuse these heterogeneous data has become a core challenge in current research. In recent years, the rapid development of deep learning technology has provided a new solution for the fusion and classification of multi-source data. In particular, convolutional neural networks (CNNs) and Transformers, as two mainstream models, have demonstrated powerful processing capabilities in multi-source remote sensing image classification tasks. Specifically, the CNN method performs outstandingly in the classification task of processing different data sources. With its excellent local spatial feature extraction ability, CNN can extract and fuse the spatial and spectral features of multi-source data layer by layer, thereby significantly improving the classification accuracy. On the other hand, Transformer can capture data associations in a global range through the self-attention mechanism, which is particularly suitable for processing high-dimensional, multi-source data such as hyperspectral images. Transformer is good at global modeling and can more comprehensively integrate feature information from different data sources to improve classification performance. In general, CNN focuses more on the extraction of local features, while Transformer achieves detailed feature fusion through global correlation modeling. Both have their own advantages in multi-source remote sensing image classification tasks.

[0004] In the field of multi-source remote sensing image classification, although deep learning technology has made significant progress in recent years, there are still some unresolved problems, which are summarized as follows: (1) Insufficient flexibility in feature extraction: Existing feature extraction methods have limitations in adapting to the frequency characteristics of different types of input data, resulting in the inability to effectively capture subtle changes in the data. (2) Poor feature fusion strategy: Existing methods fail to fully utilize the advantages of various modalities such as the spectral characteristics of hyperspectral images (HSI) and the spatial characteristics of laser radar (LiDAR) / synthetic aperture radar (SAR) during the feature fusion process, resulting in unsatisfactory fusion effects between different modalities. (3) Poor computational efficiency: Although the Transformer model performs well in capturing global features, its high computational complexity and significant computational cost limit its widespread adoption in practical applications.

[0005] In view of this, there is an urgent need to develop more flexible and efficient feature extraction and fusion methods to improve the overall performance of multi-source remote sensing image classification. Summary of the invention

[0006] In view of the problems of insufficient feature extraction flexibility, poor feature fusion strategy and poor computational efficiency in existing multi-source remote sensing image classification methods, the present invention proposes a multi-source remote sensing image classification method based on spectrum adaptive feature fusion to improve the accuracy and performance of multi-source remote sensing image classification.

[0007] To achieve the above object, the technical solution implemented by the present invention is as follows: A multi-source remote sensing image classification method based on spectrum adaptive feature fusion, comprising the following steps: S1: Preprocessing of multi-source remote sensing datasets: hyperspectral image dimension compression and data filling; after preprocessing, it is divided into training set and test set; S2: Construct a spectrum adaptive feature fusion neural network model, which includes a dynamic frequency domain feature extraction module, a spectral-spatial adaptive feature fusion module, and a fully connected layer for classification; the data processing process of the network model is represented as input data → preliminary feature extraction by convolution block → dynamic frequency domain feature extraction module → spectral-spatial adaptive feature fusion module → fully connected layer; the dynamic frequency domain feature extraction module is designed to extract frequency domain features and enhance the data expression ability through a dynamic Fourier transform filtering process; the spectral-spatial adaptive feature fusion module fuses data features from different sources, and the fully connected layer for classification is located at the last stage of the network model, and is used to classify according to the results of the previous feature extraction and fusion; S3: The spectrum adaptive feature fusion neural network model is trained using a cross entropy loss function, a weight decay term is set in the optimizer, and L2 regularization is applied to the weights to constrain the complexity of the model parameters; S4: Use the trained spectrum adaptive feature fusion neural network model to process multi-source remote sensing data and perform visual result analysis.

[0008] Furthermore, the S1 specifically includes: S1-1: Read hyperspectral image HSI and LiDAR / SAR data, and use principal component analysis (PCA) to perform dimensionality reduction on hyperspectral data; S1-2: Perform padding operations on the input hyperspectral image, the image after PCA dimensionality reduction, and the LiDAR / SAR data; S1-3: Use a window of a specified size to slide on the filled data to generate an image block corresponding to each position; S1-4: Construct a PyTorch dataset class and generate training and test sets using groundTruth labels and corresponding position indices.

[0009] Furthermore, in S2, the data processing process of the adaptive feature fusion neural network model is: S2-1: The input data is HSI and LiDAR / SAR data. The dimension of HSI input data is (B, C, N, P, P), where B is the batch size, C is the number of channels (usually 1), N is the number of hyperspectral bands, and P is the size of the cropped image block, that is, WindowSize. The input data dimension of LiDAR / SAR image is (B, C, P, P), where B is the batch size, C is the number of channels, and P is the size of the cropped image block; S2-2: The dynamic frequency domain feature extraction module is mainly responsible for extracting effective feature information from the input data: For HSI data, the data stream first undergoes preliminary feature extraction: HSI data → 3D Conv → BatchNorm3D → ReLU activation function; similarly, for LiDAR / SAR data, the data stream also undergoes preliminary feature extraction: LiDAR / SAR data → 2DConv → BatchNorm2D → ReLU activation function; the calculation process of the preliminary feature extraction operation is expressed as: ; ; in Represents the input data of HSI, Represents LiDAR / SAR data. Represents the initially extracted hyperspectral features, Represents the initially extracted LiDAR / SAR features; S2-3: For what we get and The dynamic frequency domain feature extraction module (Dynamic FourierFiltering Block) is used to extract features that adapt to the data mode. The entire structure is similar to the conventional structure of Transformer Block. The overall structure is input feature → LayerNorm → dynamic Fourier filtering → residual connection → LayerNorm → MLP → residual connection. The entire calculation process is as follows: ; ; in represents LayerNorm, It stands for Dynamic Filter. MLP is a multi-layer perceptron, and its structure is Linear→ReLU→Dropout→Linear→Dropout. The calculation process of MLP is as follows: ; The processing process is similar to processing process.

[0010] S2-3: The dynamic filter is the core module of the dynamic Fourier filter block. The filter is a learnable parameter , represents the complex field, represents the number of filters, and Represents the length and width of the input data.

[0011] ; ; ; in, Represents input data, here is or . GAP stands for global average pooling and SoftMax is used for data normalization. represents Fourier transform, is the inverse Fourier transform. is a learnable parameter, i.e., a filter, is through The generated dynamic filter kernel is multiplied by the input features after linear layer and Fourier transformation to obtain the processed features, which are then transformed into the real domain through inverse Fourier transformation and then passed through the linear layer to obtain the final output. .

[0012] S2-4: The process of data being processed by the preliminary feature extraction and dynamic frequency domain feature extraction modules will be repeated times, generally the value of L is 1 to 2. After the above feature extraction process, and The fusion process is as follows: ; ; ; ; Among them, CA stands for Channel Attention, SA stands for Spatial Attention, Represents a splicing operation, It is a commonly used activation function in deep learning. It is a weight map obtained through learning, which is used to guide the network to complement the advantages of HSI and LiDAR / SAR data to obtain better fusion features. and They represent convolutions with kernel sizes of 5 and 1 respectively.

[0013] The detailed calculation process of channel attention and spatial attention is as follows: ; ; Among them, GAP is global average pooling and GMP is global maximum pooling.

[0014] Furthermore, in S3, the model training includes:

[0015] (1) Optimizer settings. The initial learning rate is generally set to 0.0001 to control the step size of each parameter update. The choice of learning rate affects the convergence speed and stability of the model and is usually tuned according to the specific data set and task requirements. The default momentum coefficients betas = (0.9, 0.999) are used to control the first-order moment estimate (momentum) and second-order moment estimate (exponentially weighted average of the square of the gradient) of the gradient, respectively. The momentum parameter helps to suppress gradient oscillation and accelerate convergence during the optimization process. To prevent overfitting, the weight decay term weight_decay is set in the optimizer. This term constrains the complexity of the model parameters by applying L2 regularization to the weights. The numerical stability parameter eps = 1e-8 is set to avoid zero division in the denominator of the update formula, thereby enhancing the stability of the optimization process.

[0016] (2) Loss function design. In order to effectively guide the parameter update of the model, the cross entropy loss function (CrossEntropyLoss) is used as the optimization objective. The cross entropy loss function is a loss function widely used in multi-category classification tasks. Its core idea is to maximize the model's prediction probability for the correct category by measuring the difference between the predicted distribution and the true distribution. The mathematical expression of the cross entropy loss function is as follows: ; Where N is the number of samples, C is the number of categories, represents the distribution of the true label of the i-th sample in the j-th category (usually 0 or 1), It represents the predicted probability of the i-th sample in the j-th class.

[0017] (3) Forward propagation and loss calculation. The input data includes the hyperspectral image hsi_pca after dimensionality reduction and other feature data Xdata. After the input is calculated by the model, the output outputs is generated, which represents the predicted probability distribution of the input data in each category. The loss function is used to measure the difference between the predicted result of the model output and the true label, providing guidance for model optimization.

[0018] Compared with the prior art, the advantages and positive effects of the present invention are: 1. The dynamic frequency domain feature extraction module proposed in the present invention dynamically generates a global filter through fast Fourier transform, which can be flexibly adjusted according to the frequency distribution of the input features to adapt to the frequency characteristics of different images. This dynamic generation mechanism significantly enhances the flexibility and effectiveness of feature extraction. Compared with the traditional multi-head self-attention mechanism, the present invention uses fast Fourier transform to achieve efficient capture of global features. While retaining the global feature modeling capability of the multi-head self-attention mechanism, it greatly reduces the computational complexity, is particularly suitable for high-resolution image processing tasks, and greatly improves computational efficiency.

[0019] 2. The spectral-spatial adaptive fusion module of the present invention dynamically learns the fusion weights between different data sources (such as hyperspectral images and LiDAR / SAR images) by introducing an attention mechanism. Unlike the traditional simple splicing or superposition method, this module can adaptively extract and fuse the spectral features of hyperspectral images and the spatial features of LiDAR / SAR images according to the characteristics of the data. By dynamically optimizing the fusion weights, the complementarity of different data sources is fully utilized, overcoming the possible limitations of a single data source, thereby significantly improving the adaptability and classification accuracy of the model in complex tasks.

[0020] 3. The present invention not only retains the global feature capture capability but also significantly reduces the computational complexity through the collaborative design of the dynamic frequency domain feature extraction module and the spectrum-space adaptive fusion module; at the same time, through the adaptive data fusion strategy, the effective integration and feature enhancement of multi-source data are achieved. This design enables the present invention to show better performance when processing multi-source data and significantly reduces the consumption of computing resources.

[0021] The present invention realizes the classification of multi-source remote sensing data by effectively extracting and fusing the features of hyperspectral images (HSI) and laser radar (LiDAR) / synthetic aperture radar images (SAR). This technology is suitable for multi-source data fusion and classification scenarios, can cope with the remote sensing data processing needs of complex and diverse surface information, and has significant advantages in improving classification accuracy and processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1The figure is a flowchart diagram of an embodiment of the present invention.

[0023] Figure 2 It is a schematic diagram of a principle flow chart of an embodiment of the present invention.

[0024] Figure 3 Schematic diagram of the neural network structure described in an embodiment of the present invention.

[0025] Figure 4 Detailed structural diagram of the dynamic frequency domain feature extraction module according to an embodiment of the present invention.

[0026] Figure 5 Detailed structural diagram of the spectral spatial adaptive feature fusion module according to an embodiment of the present invention.

[0027] Figure 6 It is a schematic diagram of visualization images of various algorithms on the Berlin dataset according to an embodiment of the present invention.

[0028] Figure 7 It is a schematic diagram of visualization images of various algorithms on the Houston2013 dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further explained and illustrated below through specific embodiments in conjunction with the accompanying drawings.

[0030] Embodiment 1: A multi-source remote sensing image classification method based on spectrum adaptive feature fusion, referring to Figure 1 , including the following steps: (1) Preprocessing and partitioning of multi-source remote sensing datasets: (1.1) Dimension compression of hyperspectral images is the first step in multi-source remote sensing data processing. Hyperspectral images have extremely high spectral resolution, and each pixel contains multiple spectral channels. This feature enables them to accurately capture the subtle spectral features of objects. However, when the spectral dimension is too high, it will cause the computational complexity to increase sharply when used directly for training. Since hyperspectral images contain multiple spectral channels, using them directly for training will lead to a surge in computational complexity. To this end, principal component analysis (PCA) is used to reduce the dimensionality of hyperspectral data and compress the high-dimensional spectral data into fewer principal components (usually set to 30 channels). While retaining most of the key information, it significantly reduces the computational burden and improves processing efficiency.

[0031] (1.2) To ensure that the edge information is fully preserved when the sliding window extracts samples, reflection filling is performed on the hyperspectral images (original and after PCA dimensionality reduction) and LiDAR / SAR data. Reflection filling effectively expands the data based on edge features, avoids information loss, ensures data integrity and accuracy, and lays the foundation for generating high-quality samples.

[0032] (1.3) By sliding a sliding window of a specified size (usually 11 and an odd number) on the padded data, an image block sample corresponding to each position is generated, covering the original hyperspectral data, the data after PCA dimensionality reduction, and the LiDAR / SAR data. This method fully integrates multi-source data, provides rich resources for building a comprehensive and accurate data set, and facilitates the mining of potential features and patterns.

[0033] (1.4) Construct PyTorch dataset class and generate training and test sets. Construct PyTorch dataset class and generate training and test sets through groundTruth labels and corresponding position indexes. The number of samples of Berlin and Houston2013 datasets used are listed in Table 1 and Table 2 respectively. Through this construction and generation process, data can be effectively organized to meet the requirements of deep learning model training and evaluation, thereby providing reliable data support and guarantee for subsequent model training, optimization and performance evaluation.

[0034] Table 1. The number of training and testing samples of the Berlin dataset .

[0035] Table 2 Number of training and testing samples in the Houston2013 dataset .

[0036] (2) Constructing a spectrum adaptive feature fusion neural network model As shown in FIG3 , the network structure of the present invention is composed of a dynamic frequency domain feature extraction module, a spectral space adaptive feature fusion module and a classification fully connected layer. The dynamic frequency domain feature extraction module enhances the frequency domain feature expression capability of HSI and LiDAR / SAR data through the dynamic filtering process of Fourier transform; the spectral space adaptive feature fusion module utilizes the complementarity of HSI and LiDAR / SAR data to dynamically fuse the features of the two data; the classification fully connected layer is located at the end of the network and is responsible for completing the classification task based on the extracted and fused features. The flow of the overall network structure is: input data → preliminary feature extraction of convolution block → dynamic frequency domain feature extraction module → spectral space adaptive fusion module → fully connected layer. The input data includes two parts, namely HSI and LiDAR / SAR, where the input dimensions of HSI are (B, C, N, P, P), and the input dimensions of LiDAR / SAR are (B, C, P, P). Among them, B represents the batch size, C is the number of channels, N is the number of hyperspectral bands, and P is the cropped image block size (i.e., WindowSize).

[0037] (2.1) The dynamic frequency domain feature extraction module is mainly responsible for extracting effective feature information from the input data. For HSI data, the data stream first undergoes preliminary feature extraction: HSI data → 3D Conv → BatchNorm3D → ReLU activation function. Similarly, for LiDAR / SAR data, the data stream also undergoes preliminary feature extraction: LiDAR / SAR data → 2DConv → BatchNorm2D → ReLU activation function. The calculation process of the preliminary feature extraction operation can be expressed as: ; ; in Represents the input data of HSI, Represents LiDAR / SAR data. and Represents 3D and 2D batch normalization respectively. Represents the initially extracted hyperspectral features, Represents the initially extracted LiDAR / SAR features.

[0038] (2.2) If Figure 4 As shown, for the extracted and The dynamic frequency domain feature extraction module (DynamicFourier Filtering Block) is used to extract features adapted to the respective data modalities. The overall structure of this module is similar to the standard structure of the Transformer Block. The specific process is: input feature → LayerNorm → dynamic Fourier filtering → residual connection → LayerNorm → MLP → residual connection. The calculation process is as follows: ; ; in Represents LayerNorm layer normalization, It stands for Dynamic Filter. MLP is a multi-layer perceptron, and its structure is Linear→ReLU→Dropout→Linear→Dropout. The calculation process of MLP is as follows: ; The processing process is similar to processing process.

[0039] (2.3) The dynamic filter is the core module of the dynamic Fourier filter block. The filter is a learnable parameter , represents the complex field, represents the number of filters, and Represents the length and width of the input data; ; ; ; in, Represents input data, here is or . GAP stands for global average pooling and SoftMax is used for data normalization. represents Fourier transform, is the inverse Fourier transform. is a learnable parameter, i.e., a filter, is through The generated dynamic filter kernel is multiplied by the input features after linear layer and Fourier transformation to obtain the processed features, which are then transformed into the real domain through inverse Fourier transformation and then passed through the linear layer to obtain the final output. .

[0040] (2.4) If Figure 3 As shown, the process of data processing by the preliminary feature extraction and dynamic frequency domain feature extraction module will be repeated Usually, the value of L is 1 to 2. Figure 5 As shown in the figure, after the above feature extraction process, and Input the spectral space adaptive feature fusion module for fusion; the fusion process is as follows: ; ; ; ; Among them, CA stands for Channel Attention, SA stands for Spatial Attention, Represents a splicing operation, It is a commonly used activation function in deep learning. It is a weight map obtained through learning, which is used to guide the network to complement the advantages of HSI and LiDAR / SAR data to obtain better fusion features. and They represent convolutions with kernel sizes of 5 and 1 respectively.

[0041] The detailed calculation process of channel attention and spatial attention is as follows: ; ; Among them, GAP is global average pooling and GMP is global maximum pooling.

[0042] (3) Model training and quantitative evaluation (3.1) Model training process (3.1.1) Optimizer settings. The initial learning rate is usually set to 0.0001 to control the step size of each parameter update. The choice of learning rate directly affects the convergence speed and stability of the model and needs to be adjusted according to the specific data set and task requirements. The optimizer uses the default momentum coefficient betas=(0.9,0.999)betas=(0.9,0.999) to control the first-order moment estimate (momentum) and second-order moment estimate (exponential weighted average of the square of the gradient) of the gradient, respectively. The momentum parameter can effectively suppress gradient oscillation and improve the convergence efficiency of the optimization. To prevent the model from overfitting, the weight decay term weight_decay is introduced in the optimizer to impose constraints on the weights through L2 regularization to reduce the complexity of the model parameters. At the same time, the numerical stability parameter eps=1e−8 is set to avoid division by zero in the denominator of the update formula, thereby enhancing the stability and robustness of the optimization process.

[0043] (3.1.2) Loss function design. In order to effectively guide the update of model parameters, this paper adopts the cross entropy loss function (CrossEntropyLoss) as the optimization target. Cross entropy loss is a commonly used loss function for multi-category classification tasks. Its core is to optimize the model's prediction probability for the correct category by measuring the difference between the predicted distribution and the true distribution. Its mathematical expression is as follows: ; Where N is the number of samples, C is the number of categories, represents the distribution of the true label of the i-th sample in the j-th category (usually 0 or 1), It represents the predicted probability of the i-th sample in the j-th class.

[0044] (3.1.3) Forward propagation and loss calculation. The input data includes the hyperspectral image hsi_pca after dimensionality reduction and other feature data Xdata. After these data are processed by the model, the output outputs are generated, that is, the predicted probability distribution corresponding to each category. Subsequently, the loss function is used to calculate the difference between the model prediction result and the true label to provide guidance for the subsequent optimization of model parameters.

[0045] (3.2) Quantitative evaluation of the model (3.2.1) Evaluation process. In order to comprehensively evaluate the performance of the model in the classification task, the present invention has designed a systematic evaluation process. This process is carried out based on the validation data set after each training epoch, covering model prediction, performance indicator calculation and result recording, ensuring the scientificity and effectiveness of model optimization.

[0046] (3.2.2) Evaluation metrics Accuracy is an indicator that measures the proportion of samples that the model correctly classifies in each category. It is mainly used to evaluate the classification accuracy of the model. For a specific category, the accuracy rate focuses on the proportion of samples that actually belong to the category among the samples predicted by the model to be in that category. A high accuracy rate indicates that the model has a strong ability to distinguish between the categories.

[0047] Recall measures the model's ability to successfully identify true samples in each category. It reflects the model's coverage of samples in a certain category, especially when the categories are unevenly distributed. A high recall rate indicates that the model has a low rate of missed detections in a specific category.

[0048] Overall accuracy is a core indicator for evaluating the global classification ability of a model, indicating the proportion of correct classifications of the model on all test samples. It takes into account the performance of all categories and provides a direct and comprehensive measurement method.

[0049] Average precision is the arithmetic mean of the classification recall rates of all categories, which is used to measure whether the performance of the model on each category is balanced. Unlike the overall accuracy, average precision pays more attention to the performance of the model on the minority class.

[0050] The Kappa coefficient measures the consistency of the classification results by comparing the model classification results with the random classification results. The Kappa coefficient not only takes into account the classification accuracy, but also adjusts for the correct classification that may result from random guessing, so it can more objectively evaluate the classification reliability of the model.

[0051] Category-level metrics (precision, recall): These metrics can reveal the performance of the model on a specific category, help discover performance differences between categories, and are suitable for fine-grained category analysis. For example, in a garbage classification task, the model's ability to identify different types of garbage can be evaluated.

[0052] Global indicators (OA, AA, Kappa coefficient): These indicators provide an overall perspective and reflect the model's adaptability to the entire dataset. They can be used for global performance comparison and validation of the effectiveness of model improvements.

[0053] Accuracy: Accuracy measures the proportion of correctly classified samples in each category, reflecting the recognition accuracy of the model in this category. The calculation formula is as follows: ; in, : The number of correctly classified samples in the i-th class. : The number of samples in the i-th category that are misclassified as this category.

[0054] Recall: The recall rate measures the proportion of samples that are successfully identified in each category, reflecting the model's coverage of the category. The calculation formula is as follows: ; in, : The number of samples in the i-th category that are not correctly classified.

[0055] Overall Accuracy (OA): Overall accuracy measures the classification accuracy of the model for all test samples, reflecting the overall classification ability of the model. The calculation formula is as follows: ; Where, C: total number of categories. N: total number of test samples.

[0056] Average Accuracy (AA): Average accuracy is the arithmetic mean of the recall rates of each category, reflecting the average performance of the model in each category. The calculation formula is as follows: Kappa coefficient.

[0057] Kappa coefficient: The Kappa coefficient measures the consistency between the classification results and random classification and is used to evaluate the reliability of the classification results. The calculation formula is as follows: ; in, It is OA. is the expected value of the consistency of random classification, and the calculation formula is: ; Precision and recall focus on category-level evaluation, which can reveal the specific advantages or disadvantages of the model for certain categories. OA, AA, and Kappa coefficients provide a global perspective, reflecting the classification performance of the model on the entire data set.

[0058] (3.2.3) Index recording and optimal storage After each round of training, the overall accuracy (OA) is calculated based on the validation data set and recorded for subsequent performance comparison and model preservation. Subsequently, the OA value of the current validation set is compared with the saved historical optimal OA value: if the current OA value exceeds the historical optimal value, the optimal OA is updated and the current model is saved; otherwise, the historical optimal value remains unchanged.

[0059] (3.2.4) Termination conditions If the optimal OA value of the validation set does not improve in 20 consecutive epochs, the model performance is considered to have stabilized. In this case, the training process will be automatically terminated to avoid unnecessary computational overhead and improve training efficiency.

[0060] (4) Model visualization results analysis This study visualized the prediction results of the classification model and generated intuitive color classification images to analyze the classification effect and spatial distribution characteristics of the model. Input data processing: The model is used to infer the input data (such as hyperspectral images and LiDAR / SAR data) to generate the category prediction results for each pixel. Result mapping: According to the category color mapping rules of different data sets (such as Berlin_color_map, Houston2013_color_map, etc.), the predicted category is mapped to the corresponding color value to generate a color image. Image generation and saving: By updating the image matrix pixel by pixel, OpenCV is finally used to save the generated color image as a .png file. The file name is named with the dataset name and the current timestamp to ensure the uniqueness of the file. The distribution of classification results is directly displayed through color images, providing an intuitive basis for model performance evaluation.

[0061] Embodiment 2: This embodiment further illustrates the effect of the present invention in combination with simulation experiments:

[0062] The simulation experiment of this embodiment is carried out in the hardware environment of Intel(R) Core(TM) i9-10920X CPU, NVIDIA GeForceRTX 4090, 32GB memory and the software environment of Ubuntu 18.04.6 LTS, Python 3.10.13, Torch 2.1.1+cu118. The simulation experiment data of the present invention are Berlin data set and Houston2013 data set. Houston2013 data set, as part of the 2013 IEEE Geoscience and Remote Sensing Society Data Fusion Competition, provides a unique perspective to observe the urban land cover in Houston, Texas and its surrounding areas. The data set records 144 bands of hyperspectral images covering the spectral range of 380 to 1050 nanometers at a spatial resolution of 2.5 meters, as well as LiDAR data that provides accurate altitude information, covering 15 different land cover types. The Berlin data set covers urban and rural areas of Berlin. Hyperspectral data is environmental mapping and analysis program (EnMAP) data simulated based on hyperspectral imager (HyMap) data. Synthetic aperture radar (SAR) data are obtained from the European Space Agency (ESA) Sentinel-1 dual-polarization (VV-VH) single-look complex (SLC) product. The hyperspectral image (HSI) has 797 × 220 pixels and 244 spectral bands in the wavelength range of 400–2500 nm (reference [berlin]). The SAR image has 1723 × 476 pixels. The nearest neighbor interpolation method was used to match the spatial resolution of the hyperspectral image and SAR data.

[0063] The method of the present invention is compared with the existing advanced multi-source remote sensing image classification methods. The TBCNN method used in the comparative experiment is proposed in the article "Multisource remote sensing data classification based on convolutional neural network"; the FusAtNet method is proposed in the article "Fusatnet: Dualattention based spectrospatial multimodal fusion network for hyperspectraland lidar classification"; The method was proposed in the article "S2enet: Spatial–spectralcross-modal enhancement network for classification of hyperspectral and lidardata"; the DFINet method was proposed in the article "Hyperspectral and multispectral classificationfor coastal wetland using depthwise feature interaction network"; the ExViT method was proposed in the article "Extended Vision Transformer (ExViT) for Land Use and LandCover Classification: A Multimodal Deep Learning Framework"; the HCT method was proposed in the article "Joint classification of hyperspectral and LiDAR data using ahierarchical CNN and Transformer".

[0064] Depend on Figure 6 It can be seen that in the processing of the Berlin dataset, the classification map generated by the method used in the present invention is highly consistent with the actual situation. Compared with other methods that frequently make classification errors in industrial areas and areas with low plant distribution, the method of the present invention shows excellent accuracy and can accurately classify these complex and challenging areas, especially in retaining subtle textures and detailed features, making the classification results more accurate and reliable. Figure 7 It can be seen that when applied to the Houston 2013 dataset, the method of the present invention achieves a more accurate segmentation effect, effectively reduces noise interference in most areas, and significantly improves the clarity and accuracy of the overall results, providing a better data foundation and decision-making basis for the analysis and application of the dataset.

[0065] Tables 3 and 4 show the comparison of various indicators between the proposed method and other methods under the Berlin dataset and the Houston 2013 dataset, respectively. It can be clearly seen that the proposed method has achieved the best level in the three key indicators of overall accuracy (OA), average accuracy (AA) and Kappa coefficient. Specifically, in the application of the Berlin dataset, the proposed method has shown better results than other methods in the classification of specific landform types such as industrial areas, low vegetation areas and water areas, and can more accurately identify and divide these areas, thereby significantly improving the accuracy and reliability of classification. In the Houston 2013 dataset, the proposed method also outperforms other methods in the classification of healthy grassland, stressed grassland, commercial area, highway and parking lot 2, effectively reducing the probability of misclassification and omission, providing strong support for the accurate classification and in-depth analysis of the dataset, and further highlighting the advantages and value of the proposed method in the classification task of complex datasets.

[0066] Table 3 Comparison of performance indicators of various algorithms on the Berlin dataset

[0067] .

[0068] Table 4 Comparison of performance indicators of various algorithms on the Houston2013 dataset .

[0069] The multi-source remote sensing image classification method based on spectrum adaptive feature fusion provided by the present invention is mainly to accurately identify the complex land object types covered by the multi-source remote sensing image data set, and to adaptively extract and fuse the spectrum features of remote sensing data from different sources, so as to effectively improve the accuracy and reliability of image classification, thereby meeting the needs of professional fields such as high-precision urban planning, land use monitoring, and environmental resource assessment, but it is also applicable to other multi-source remote sensing image classification scenarios with similar characteristics, such as fine classification of agricultural production areas to assist precision agricultural management, ecosystem monitoring of nature reserves to achieve biodiversity protection, and rapid assessment of disaster areas to support emergency rescue decisions, etc. In these different application scenarios, the beneficial effects achieved are also similar.

[0070] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A multi-source remote sensing image classification method based on spectrum adaptive feature fusion, characterized in that: The method comprises the following steps: S1: Preprocessing of multi-source remote sensing datasets: hyperspectral image dimension compression and data filling; after preprocessing, it is divided into training set and test set; S2: Construct a spectrum adaptive feature fusion neural network model, which includes a dynamic frequency domain feature extraction module, a spectrum space adaptive feature fusion module and a fully connected layer for classification; the data processing process of the network model is represented as input data → preliminary feature extraction by convolution block → dynamic frequency domain feature extraction module → spectrum space adaptive feature fusion module → fully connected layer; the dynamic frequency domain feature extraction module is designed to extract frequency domain features and enhance the data expression ability through a dynamic Fourier transform filtering process; the spectrum space adaptive feature fusion module fuses data features from different sources, and the fully connected layer for classification is located at the last stage of the network model, and is used to perform classification according to the results of the previous feature extraction and fusion; S3: The spectrum adaptive feature fusion neural network model is trained using a cross entropy loss function, a weight decay term is set in the optimizer, and L2 regularization is applied to the weights to constrain the complexity of the model parameters; S4: Use the trained spectrum adaptive feature fusion neural network model to process multi-source remote sensing data and perform visual result analysis.

2. The multi-source remote sensing image classification method according to claim 1, characterized in that: The S1 specifically includes: S1-1: Read hyperspectral image HSI and LiDAR / SAR data, and use principal component analysis (PCA) to perform dimensionality reduction on hyperspectral data; S1-2: Perform padding operations on the input hyperspectral image, the image after PCA dimensionality reduction, and the LiDAR / SAR data; S1-3: Use a window of a specified size to slide on the filled data to generate an image block corresponding to each position; S1-4: Construct a PyTorch dataset class and generate training and test sets using groundTruth labels and corresponding position indices.

3. The multi-source remote sensing image classification method according to claim 1, characterized in that: In S2, the data processing process of the adaptive feature fusion neural network model is: S2-1: The dimension of the input data HSI is (B, C, N, P, P), where B is the batch size, C is the number of channels, N is the number of hyperspectral bands, and P is the size of the cropped image block, i.e., WindowSize; the input data dimension of the input data LiDAR / SAR image is (B, C, P, P), where B is the batch size, C is the number of channels, and P is the size of the cropped image block; S2-2: For HSI data, the data stream first undergoes preliminary feature extraction: HSI data → 3D Conv → BatchNorm3D → ReLU activation function; for LiDAR / SAR data, the data stream undergoes preliminary feature extraction: LiDAR / SAR data → 2DConv → BatchNorm2D → ReLU activation function; the calculation process of the preliminary feature extraction operation is expressed as: ； ； in Represents the input data of HSI, Represents LiDAR / SAR data; Represents the initially extracted hyperspectral features, Represents the initially extracted LiDAR / SAR features; S2-3: For what we get and The dynamic frequency domain feature extraction module is used to extract features that adapt to the data modality. The overall structure is input feature → LayerNorm → dynamic Fourier filter → residual connection → LayerNorm → MLP → residual connection; the entire calculation process is as follows: ； ； in represents LayerNorm, The representative dynamic filter MLP is a multi-layer perceptron, and its structure is Linear→ReLU→Dropout→Linear→Dropout; the calculation process of MLP is as follows: ； The processing process is similar to The processing process; S2-4: Filters are learnable parameters , represents the complex field, represents the number of filters, and Represents the length and width of the input data; ； ； ； in, Represents input data, here is or , GAP stands for global average pooling, and SoftMax is used for data normalization; represents Fourier transform, is the inverse Fourier transform, is a learnable parameter, is through The generated dynamic filter kernel is multiplied by the input features after linear layer and Fourier transformation to obtain the processed features, which are then transformed into the real domain through inverse Fourier transformation and then passed through the linear layer to obtain the final output. ; S2-5: The process of data being processed by the preliminary feature extraction and dynamic frequency domain feature extraction modules will be repeated Second, and The spectral spatial adaptive feature fusion module is used for fusion. The fusion process is as follows: ； ； ； ； Among them, CA stands for channel attention, SA stands for spatial attention, Represents a splicing operation, It is a commonly used activation function in deep learning. is the weight map obtained through learning, and They represent convolutions with kernel sizes of 5 and 1 respectively; the detailed calculation process of channel attention and spatial attention is as follows: ； ； Among them, GAP is global average pooling and GMP is global maximum pooling.

4. The multi-source remote sensing image classification method according to claim 1, characterized in that: In S3, the model training includes: (1) Optimizer settings: The learning rate is tuned according to the specific data set and task requirements, and the default momentum coefficient is used to control the first-order moment estimation and second-order moment estimation of the gradient respectively. To prevent overfitting, a weight decay term is set in the optimizer, and the complexity of the model parameters is constrained by applying L2 regularization to the weights. (2) The cross entropy loss function is used as the optimization objective, and the mathematical expression is as follows: ； Where N is the number of samples, C is the number of categories, represents the distribution of the true label of the i-th sample in the j-th category, Represents the predicted probability of the i-th sample in the j-th category; (3) Forward propagation and loss calculation: After the input data is calculated by the model, outputs are generated, which represent the predicted probability distribution of the input data in each category.

Images