Hyperspectral image classification method based on multi-modal data

Abstract

The invention relates to the field of image classification, in particular to a hyperspectral image classification method based on multi-modal data. The image classification method mainly comprises the following steps: constructing a framework consisting of the feature extraction network, the filtering encoder and the joint decoder; respectively carrying out abstract feature extraction and serialization on the hyperspectral image, the Lidar data corresponding to the hyperspectral image and / or the pseudo RGB image based on a convolutional neural network; filtering the extracted features by using self-enhancement attention and filtering attention to obtain filtered information, and filtering out interference terms which hinder information fusion; and fusing the filtered information by using a joint decoder and predicting a final classification result. Compared with the existing image classification method, the image classification method can effectively utilize various information sources to improve the classification performance of the hyperspectral image, and solves the problem that the existing multi-modal data lacks the flexibility of expanding to other modals during application.

Description

Technical Field

[0001] The present invention relates to the field of image classification, and in particular to a hyperspectral image classification method based on multimodal data. Background Art

[0002] Regarding hyperspectral image classification, the existing technologies mainly include traditional machine learning methods, convolutional neural network-based methods, Transformer-based methods and multimodal-based methods.

[0003] Traditional machine learning methods for hyperspectral image classification usually rely on hand-crafted features such as texture, color, spectral and spatial details, as well as various classification techniques. Commonly used feature extraction methods include Histogram of Oriented Gradients (HOG) and Scale-Invariant Feature Transform (SIFT). After feature extraction, classification algorithms such as Support Vector Machines (SVM) and k-nearest Neighbors (KNN) are used for classification. However, due to the limitations of hand-crafted features, these methods face challenges when applied to real-world data. Therefore, traditional hyperspectral image classification methods struggle with feature selection and extraction, and lack the utilization and adaptation of information.

[0004] Convolutional Neural Networks (CNNs) achieve automatic feature learning through convolutional layers and use fully connected layers for classification, which can overcome the limitations of traditional methods and improve classification accuracy by capturing the complex spatial-spectral relationship in hyperspectral data. Although some progress has been made in hyperspectral image classification by combining early deep learning techniques such as convolutional neural networks, these methods focus on local information and have difficulty modeling the relationship between feature nodes.

[0005] Unlike convolutional neural networks (CNNs), Transformers rely on self-attention to capture the dependencies between different elements in a sequence, thereby effectively modeling long-range dependencies. The introduction of Vision Transformers (ViT) has promoted its application in the field of hyperspectral image classification. Transformers can effectively capture the dependencies between spectral bands and spatial positions in hyperspectral images. The architecture generates a contextual representation for each pixel in a hyperspectral image. By considering the entire image context, the model can learn to encode information about neighboring pixels and their categories, thereby improving classification accuracy. After extracting spatial and spectral features using spatial-spectral correlation contrastive learning, spatial and spectral Transformer modules are used to learn high-level semantic features, and spatial-spectral enhancement and image entropy are used to generate label information. At the same time, a 2D convolution module for dimensionality reduction, a stacked CNN for preserving spectral distribution, and a Transformer self-attention mechanism for capturing long-term relationships are integrated. Despite continuous improvements in the structure, existing Transformer-based methods still have difficulty in obtaining hyperspectral data. Limited data limits performance improvements, and it is also difficult to capture local spatial-spectral fusion information.

[0006] Despite the continuous improvement of model structure and task performance, it is challenging to obtain a large amount of hyperspectral data with limited information. Therefore, further improving the accuracy has become a daunting task, and many works have begun to consider combining data from other modalities to utilize more information. The Generative Adversarial Network (GAN) module and the interactive GAN module are introduced to learn perturbation-insensitive feature representations and eliminate the gap between different modalities, respectively, resulting in more effective and robust information transfer; CCR-Net learns the fusion representation of different data sources through cross-modal reconstruction, thereby achieving more effective information exchange between different modalities; the basic model adaptation (FMA) framework for multimodal remote sensing data classification includes a cross-spatial interaction module and a cross-channel interaction module to extract unimodal features along the spatial and channel dimensions, respectively; a U-Net feature extraction backbone network is used to learn multi-scale insensitive features from multimodal data. The MTFFN transfers the bimodal knowledge from the natural image domain and the hyperspectral image domain to the target domain through two branches, and then adopts a multi-task learning strategy to fuse the features and generate the final classification results; SMDN uses an unsupervised encoder-decoder structure as the backbone to build a multimodal architecture through joint training with data-specific branches. In order to obtain discriminative multimodal representations, it uses limited labeled samples to guide the collaborative training of two unsupervised features mapped in the latent subspace. Although many research works have integrated multimodal data to improve performance, they often involve complex models or adaptation to specific modalities and lack a unified classification algorithm framework. Summary of the invention

[0007] The purpose of the present invention is to overcome the problem of lack of a unified classification algorithm framework when using multimodal data in the prior art, and to provide a hyperspectral image classification method based on multimodal data.

[0008] In a first aspect, the present invention provides a hyperspectral image classification method based on multimodal data, characterized in that it comprises the following steps:

[0009] S1. Performing feature extraction on the hyperspectral image, and performing feature extraction on the hyperspectral image, the Lidar data corresponding to the hyperspectral image, and / or the pseudo RGB image respectively;

[0010] S2, filtering the features extracted in S1 to obtain filtered information, and filtering out interference items that hinder information fusion;

[0011] S3. Use a joint decoder to fuse the filtered information and predict the final classification result.

[0012] Preferably, the data needs to be preprocessed before step S1, mainly to enhance feature expression by reducing the dimension to reveal the intrinsic characteristics of the data. Specifically, the hyperspectral image is represented as LiDAR data is represented as Where W and H represent the spatial width and height, and B represents the number of spectral bands; a single pixel in the hyperspectral image can be defined as p i,j ∈I H , where i = 1, ..., W, j = 1, ..., H, each pixel contains B spectral bands; use principal component analysis to generate a pseudo RGB input image and obtain pseudo RGB data I R ; From the hyperspectral image data I H Get a spectrum-space cube From the pseudo RGB data I R Get an RGB cube from From the LiDAR data I L Extract a LiDAR data slice from

[0013] Preferably, the feature extraction in step S1 is to extract and serialize abstract features of the hyperspectral image, the Lidar data corresponding to the hyperspectral image and / or the pseudo RGB image respectively.

[0014] Further preferably, the abstract feature extraction specifically includes the deep features and multimodal features extracted by the backbone, specifically, 2D convolution is used after 3D convolution and before Transformer, and the process of abstract feature extraction can be defined as follows:

[0015]

[0016]

[0017] Where: φ(I 玎 ) represents the hyperspectral image, φ(I L ) represents the LiDAR data, φ(I R ) represents the pseudo RGB data, is the deep feature extracted by the backbone, F(I L , I R ) is the multimodal feature, Conv2D is 2D convolution, and Conv3D is 3D convolution.

[0018] Further preferably, the serialization needs to be performed after the abstract feature extraction, and the specific steps are as follows:

[0019] S11, serializing the deep features extracted from the backbone is to convert the deep features extracted from the backbone into a tokenized form, using the four different feedforward neural networks, and the specific process is as follows:

[0020] X′ o =X′ f2 *X f1

[0021] X′ f2 =Softmax(X f2 )

[0022]

[0023]

[0024] Where: X′ o is the tokenized form of the deep features extracted by the backbone, X′ f2 Yes X f2 The result obtained after processing by the Softmax function, X f1 , X f2 is the feedforward neural network output of HIS, W f1 and b f1 are the learnable weights and biases of the first feedforward neural network, W f2 and b f2 are the learnable weights and biases of the second feedforward neural network, represents the deep features extracted by the backbone;

[0025] S12, serializing the multimodal features is to convert the multimodal features into a tokenized form, using the four different feedforward neural networks, and the specific process is as follows:

[0026] Y o =Y f2 *Y f1

[0027] Y′ f2 =Softmax(Y f2 )

[0028] Y f1 =F(I L , I R )·W f3 +b f3

[0029] Y f2 =F(I L , I R )·W f4 +bf4

[0030] Where: Y′ o is the tokenized form of the multimodal feature after transformation, Y′ f2 Yes f2 The result obtained after processing by the Softmax function, Y f1 , Y f2 is the feedforward neural network output of HIS, W f3 and b f3 are the learnable weights and biases of the third feedforward neural network, W f4 and b f4 are the learnable weights and biases of the fourth feedforward neural network, F(I L , I R ) is the multimodal feature;

[0031] Preferably, the step S2 utilizes the self-enhanced attention and the filtered attention to filter out interference items in each modality that hinder information fusion.

[0032] Further preferably, the self-enhanced attention is used to capture the differences within a single modality, specifically using the Transformers shared attention mechanism, which can distinguish the importance of the single modality information within the abstract feature, and the formula is as follows:

[0033]

[0034] Where: MultiHeadAttn QKV is the result calculated using the attention mechanism shared by the Transformers, where Q, K, and V represent query, key, and value, respectively. k The dimension representing the key value;

[0035] Further preferably, the filtering attention selects the abstract features from different modalities to form queries and key values ​​and calculates scores to measure the relationship between different modalities. Low scores can adaptively filter out interference items, and high scores can enhance the most beneficial information for classification. o , Y′ o ), the detailed calculation in the hyperspectral image branch can be expressed as follows:

[0036]

[0037]

[0038]

[0039] Where: is X′ o and Y′ o The transformed dimensionality-increasing feature, X′ o is the tokenized form of the deep features extracted by the backbone, Y′ o is the tokenized form of the multimodal feature after transformation, FFN is the front neural network, LN is the layer normalization operation, FA is the filter attention, MLP is a two-layer linear projection, is the result obtained after being processed by the pre-neural network, is the result obtained after the filtering attention processing, is a modal feature.

[0040] Preferably, the step S3 needs to enhance the information fusion capability between all modalities and perform joint decoding, and the specific steps are as follows:

[0041] S31. Connect all the modal features and set the first token as CLS token. This process is described as follows:

[0042]

[0043] Where: is the fused sequence, fusion feature The query, key and value as self-enhanced attention mechanism realizes attention fusion of all modalities, is the modal feature, is Y′ o Change the query to X′ o Changing to said modal features calculated from keys and values;

[0044] S32. Use multilayer perceptron as the final prediction model for classification and use cross entropy loss for classification tasks. The specific process is as follows:

[0045]

[0046] Where: is the final specific classification result, yi is the prediction after the self-enhanced attention and the multi-layer perceptron processing, λ i is a hyperparameter.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention provides a hyperspectral image classification method based on multimodal data. By constructing a framework consisting of the feature extraction network, the filter encoder and the joint decoder, it is possible to effectively utilize multiple information sources to improve the hyperspectral image classification performance, while solving the problem that the current multimodal data lacks the flexibility to be extended to other modalities when applied. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 1 is a method step diagram of a hyperspectral image classification framework based on multimodal data in an embodiment of the present invention.

[0050] Figure 2 Schematic diagram of the feature extractor and serialization module in an embodiment of the present invention.

[0051] Figure 3 This is an example diagram of the Houston data set in an embodiment of the present invention.

[0052] Figure 4 This is a graph showing the comparison of the MIFF algorithm in an embodiment of the present invention with other methods on a dataset of the University of Houston using only HS images.

[0053] Figure 5 This is a result diagram of the comparison between the MIFF algorithm in an embodiment of the present invention and other methods on a dataset of the University of Trento that only uses HS images.

[0054] Figure 6 This is a graph showing the comparison results of the MIFF algorithm in an embodiment of the present invention with other methods on a dataset of MUUFL University using only hs images.

[0055] Figure 7 This is a result diagram comparing the MIFF algorithm with other methods on the Houston multi-source dataset in an embodiment of the present invention.

[0056] Figure 8 This is a result diagram comparing the MIFF algorithm with other methods on the Trento multi-source dataset in an embodiment of the present invention.

[0057] Fig. 9 This is a result diagram comparing the MIFF algorithm with other methods on the MUUFL multi-source dataset in an embodiment of the present invention.

[0058] Fig.10 Graph showing the ablation experiment results of learnable serialization in an embodiment of the present invention.

[0059] Fig.11 1 is a diagram showing comparative experimental results of the filter encoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0061] Example 1

[0062] The purpose of the present invention is to overcome the problem in the prior art of lacking a unified classification algorithm framework when using multimodal data, and to provide a hyperspectral image classification method based on multimodal data.

[0063] The present invention adopts three configurations in Example 1, namely, training and testing configuration, data set configuration and evaluation configuration.

[0064] Training and testing configuration: All training and testing processes were performed on Python 3.8 and PyTorch 1.5 framework based on Ubuntu 20.04.4 LTS system. The model of the present invention was trained and tested on Nvidia RTX 3090 GPU, which has 24GB of video memory, 19.5GHz memory frequency, 384-bit width and 936GB / s bandwidth. Training was performed on Houston, Trento and MUUFL datasets for 100 rounds with a batch size of 64. The Adam optimizer was used for optimization, with an equal step descent learning strategy, a parameter factor gamma of 0.9, and a learning rate adjusted from 5e-3 to 5e-4 at the 20th round. The training and testing experiments were repeated three times for each dataset, and the average performance of the three times was reported. During testing, the batch size was set to 500. The size of the data blocks cropped from hyperspectral images and other modalities was set to 11×11.

[0065] Dataset configuration: Three different hyperspectral images and their associated LiDAR images are used, and pseudo RGB spatial images are generated through PCA to evaluate the performance of the proposed filtering-based multimodal unified architecture. The experimental datasets include Houston, Trento, and MUUFL. The following is a detailed introduction to the above datasets:

[0066] 1. The Houston dataset is a multimodal hyperspectral image dataset with a LiDAR source, all with a spatial resolution of 2.5m. The hyperspectral images collected by the Compact Airborne Spectral Imager were provided by the IEEE Geoscience and Remote Sensing Society for its data fusion competition in 2013. They consist of 144 spectral bands in the 380nm to 1050nm region and are calibrated to the spectral radiance units of the sensor. The image size is 340×1905 pixels, covering a wavelength of 0.38-1.05um. It includes 15 different land cover categories, including healthy grass, synthetic grass, trees, soil, water, residential, commercial, roads, highways, railways, parking lots, tennis courts, and runways. Examples of this dataset include Figure 3 shown.

[0067] 2. The Trento dataset is also a multimodal hyperspectral image dataset, including hyperspectral images and LiDAR data collected in the rural area of ​​southern Trento, Italy. The AISA Eagle sensor acquires hyperspectral image data with a spatial resolution of 1m, covering 600×166 pixels, 63 spectral bands, and a spectral range of 0.42-0.99um. The Optech ALTM3100EA sensor is used to acquire LiDAR data. The dataset includes 6 types of objects: apple trees, buildings, ground, woods, vineyards, and roads.

[0068] 3. The MUUFL dataset is a multimodal hyperspectral image dataset collected in November 2010 at the Gulf Park campus of the University of Southern Mississippi in Mississippi, USA. Collected using the ITRES CASI-1500 sensor, the hyperspectral image data has a spatial resolution of 0.54×1.0m, consists of 325×220 pixels, 72 spectral channels (64 available bands), ranging from 375 to 1050nm, and a bandwidth of 10nm. In addition, the dataset also contains LiDAR elevation data for 2 rasters. Due to noise, the first and last 8 bands were removed to obtain 64 available bands. The dataset includes 11 types of ground features, such as road material, pure grass, mixed ground, and cloth panels.

[0069] Evaluation indicators: In order to evaluate the performance of the network and compare it with other methods, three popular quantitative indicators are used: overall accuracy (OA), average accuracy (AA) and statistical Kappa coefficient. OA represents the proportion of correct samples in all samples, and AA represents the average accuracy. The Kappa value reflects the degree of consistency between the classification result and the provided ground truth. The Kappa formula is:

[0070]

[0071]

[0072] Where X ii are the items on the diagonal of the confusion matrix, X ir is the observation in the i-th row, X ic is the observation in column i. Then, p e represents the proportion of observed accuracy, indicating correctly classified samples, and p0 represents the proportion of units with accidental consistency, indicating misclassification due to random factors. In practical applications, there will be imbalanced data sets, so Kappa is used to reflect this impact.

[0073] A hyperspectral image classification method based on multimodal data, the flow chart is as follows Figure 1 As shown, it is characterized in that it includes the following steps:

[0074] S1. Performing feature extraction on the hyperspectral image, and performing feature extraction on the hyperspectral image, the Lidar data corresponding to the hyperspectral image, and / or the pseudo RGB image respectively;

[0075] S2, filtering the features extracted in S1 to obtain filtered information, and filtering out interference items that hinder information fusion;

[0076] S3. Use a joint decoder to fuse the filtered information and predict the final classification result.

[0077] The feature extraction in step S1 is to extract and serialize abstract features of the hyperspectral image, the Lidar data corresponding to the hyperspectral image, and / or the pseudo RGB image.

[0078] The abstract feature extraction in step S1 specifically includes the deep features and multimodal features extracted by the backbone, specifically using 2D convolution after 3D convolution and before Transformer. The process of abstract feature extraction can be defined as follows:

[0079]

[0080]

[0081] Where: φ(I 玎 ) represents the hyperspectral image, φ(I L ) represents the LiDAR data, φ(I R ) represents the pseudo RGB data, is the deep feature extracted by the backbone, F(I L , I R ) is the multimodal feature, Conv2D is 2D convolution, and Conv3D is 3D convolution.

[0082] After the abstract feature extraction in step S1, serialization needs to be performed. The specific steps are as follows:

[0083] S11, serializing the deep features extracted from the backbone is to convert the deep features extracted from the backbone into a tokenized form, using the four different feedforward neural networks, and the specific process is as follows:

[0084] X′ o =X′ f2 *X f1

[0085] X′ f2 =Softmax(X f2 )

[0086]

[0087]

[0088] Where: X′ o is the tokenized form of the deep features extracted by the backbone, X′ f2 Yes X f2 The result obtained after processing by the Softmax function, X f1 , X f2 is the feedforward neural network output of HIS, W f1 and b f1 are the learnable weights and biases of the first feedforward neural network, W f2 and b f2 are the learnable weights and biases of the second feedforward neural network, represents the deep features extracted by the backbone;

[0089] S12, serializing the multimodal features is to convert the multimodal features into a tokenized form, using the

[0090] Four different feedforward neural networks, the specific process is as follows:

[0091] Y′ o =Y′ f2 *Y f1

[0092] Y′ f2 =Softmax(Y f2 )

[0093] Y f1 =F(I L , I R )·W f3 +b f3

[0094] Y f2 =F(I L , I R )·W f4 +b f4

[0095] Where: Y′ o is the tokenized form of the multimodal feature after transformation, Y′ f2 Yes f2 The result obtained after processing by the Softmax function, Y f1 , Y f2 is the feedforward neural network output of HIS, W f3 and b f3 are the learnable weights and biases of the third feedforward neural network, W f4 and b f4 are the learnable weights and biases of the fourth feedforward neural network, F(I L , I R ) is the multimodal feature;

[0096] Step S2 uses the self-enhanced attention and the filtered attention to filter out interference items in each modality that hinder information fusion.

[0097] The self-enhanced attention in step S2 is used to capture the differences within a single modality. Specifically, the Transformers shared attention mechanism is used, which can distinguish the importance of the single modality information within the abstract feature. The formula is as follows:

[0098]

[0099] Where: MultiHeadAttn QKV is the result calculated using the attention mechanism shared by the Transformers, where Q, K, and V represent query, key, and value, respectively. k The dimension representing the key value;

[0100] The filtering attention in step S2 is to select the abstract features from different modalities to form the query and key values ​​and calculate the scores to measure the relationship between different modalities. Low scores can adaptively filter out interference items, and high scores can enhance the most beneficial information for classification. o , Y′ o ), the detailed calculation in the hyperspectral image branch can be expressed as follows:

[0101]

[0102]

[0103]

[0104] Where: is X′ o and Y′ o The transformed dimensionality-increasing feature, X′ o is the tokenized form of the deep features extracted by the backbone, Y′ o is the tokenized form of the multimodal feature after transformation, FFN is the front neural network, LN is the layer normalization operation, FA is the filter attention, MLP is a two-layer linear projection, is the result obtained after being processed by the pre-neural network, is the result obtained after the filtering attention processing, is a modal feature.

[0105] Step S3 needs to enhance the information fusion capability between all modalities and perform joint decoding. The specific steps are as follows:

[0106] S31. Connect all the modal features and set the first token as CLS token. This process is described as follows:

[0107]

[0108] Where: is the fused sequence, fusion feature The query, key and value as self-enhanced attention mechanism realizes attention fusion of all modalities, is the modal feature, It's Y' o Change the query to X' o Changing to said modal features calculated from keys and values;

[0109] S32. Use multilayer perceptron as the final prediction model for classification and use cross entropy loss for classification tasks. The specific process is as follows:

[0110]

[0111] Where: is the final specific classification result, y i is the prediction after the self-enhanced attention and the multi-layer perceptron processing, λ i is a hyperparameter.

[0112] The final calculation results are shown below: Figure 4 , Figure 5 and Figure 6To compare the OA, AA and Kappa indicators using only the single-modality data of hyperspectral images; Figure 7 , Figure 8 and Fig. 9 The corresponding indicators of multiple modal information are compared, including LiDAR and pseudo RGB data; the best results are highlighted in bold. The experimental results show that the proposed method achieves the highest OA, AA and Kappa, which is significantly better than other methods. In general, the Transformer network-based method surpasses the traditional KNN and CNN methods, while the proposed method surpasses the current Transformer-based method.

[0113] The model proposed in this invention performs well in terms of accuracy, surpassing other methods in the OA, AA and Kappa indicators of the three datasets. Due to the improvements in the unified multimodal data source and the filtering Transformer structure, Example 1 shows that the single modality ( Figure 4 , Figure 5 and Figure 6 ) and multimodal ( Figure 7 , Figure 8 and Fig. 9 The default result contains the attention structure proposed by the present invention, and its function is described in detail in the ablation section.

[0114] Combining hyperspectral image data with LiDAR data can improve the performance of most models. The more data sources are integrated, the better the results are. The evaluation clearly shows that the proposed method uses multi-source data and Transformer structure to achieve significant performance improvement over the existing technology, such as Figure 4 As shown in Figure 2, among the traditional methods including SVM, CNN and RNN, RF has the best performance, with average OA, AA and Kappa of 74.87%, 77.94% and 72.93% respectively. However, compared with the current Transformer-based algorithms, RF shows significant differences in these indicators.

[0115] The unified multimodal Transformer network proposed in this paper outperforms other Transformer-based networks, such as ViT and Spectral Former, with average OA, AA and Kappa of 87.10%, 88.70% and 86.00% respectively. This excellent performance is mainly due to the fact that the model of this invention can adaptively filter out interference information in hyperspectral images, thus surpassing ViT and SpectralFormer without this function.

[0116] Traditional methods such as convolutional neural networks (CNN) and recurrent neural networks (RNN) rely only on convolution kernels to capture semantic information and cannot fully capture spatial relationships. This limitation leads to its poor performance compared to the Transformer network. However, applying only the Transformer, as in ViT, will include a lot of interference information due to over-considering irrelevant information. The method proposed in this paper solves this problem by effectively filtering out interference information, effectively improving the classification accuracy. Table 2 shows the performance of the proposed method compared with the Transformer network. Figure 4 Different trends. CNN3D outperforms not only CNN2D, RNN and SVM, but also ViT and SpectralFormer (SF). This shows that using only the Transformer structure may be harmful in scenes with distractors. By using the new Transformer-based filtering technology, the model of the present invention significantly improves the performance, with average OA, AA and Kappa reaching 97.15%, 95.49% and 96.18% respectively, which is a significant improvement of 2.6%, 4.5% and 10.3% compared with ViT.

[0117] In the MUUFL dataset, RF performs best among traditional methods such as CNN2D, RNN and SVM, but still lags behind Transformer-based algorithms. This shows the advantage of Transformer structures in most scenarios because they are able to consider long-term relationships. It is worth noting that the method of the present invention achieves the best performance, with average OA, AA and Kappa of 92.68%, 77.82% and 90.33% respectively.

[0118] Figure 7 It is shown that the accuracy of all classical networks in the Houston dataset is greatly improved when considering multimodal inputs (including hyperspectral, pseudo-RGB and LiDAR images). In addition to the method of the present invention (MIFF), other methods are also trained and evaluated by fusing hyperspectral and LiDAR images. The present invention represents the input source combinations of the model as MIFFHL (hyperspectral + LiDAR), MIFFHI (hyperspectral + Pseudo-Pseudo-RGB) and MIFFHLI (hyperspectral + LiDAR + Pseudo-RGB).

[0119] The proposed method consistently outperforms the other models, among which MIFFHLI, which combines three modalities, achieves the best results with average OA, AA and Kappa of 90.02%, 91.26% and 89.17%, respectively. MIFFHI has relatively lower performance than the other two fusion methods. It is worth noting that MIFFHL performs better in certain land use types. This shows the effectiveness of the proposed model in utilizing multi-source data to achieve higher classification accuracy.

[0120] Figure 8 Shows the Figure 7 Similar trends, only ViT achieves 100% accuracy in Trento’s land use type 4. MIFFHLI still achieves the best performance, with OA, AA, and Kappa of 98.63%, 97.43%, and 98.16%, respectively. It is followed by MIFFHL, with OA, AA, and Kappa of 98.16%, 94.59%, and 97.54%, respectively.

[0121] Fig. 9 The performance results of the MUUFL dataset are shown. The best performance is MIFFHLI, with OA, AA and Kappa of 93.53%, 78.91% and 91.45% respectively. The second best performance is MIFFHL, with OA, AA and Kappa of 93.17%, 77.52% and 90.92% respectively. Figure 8 Compared with the differences observed in Fig. 9 The performance gap between MIFFHLI and MIFFHL is smaller, which shows that the advantages of the model proposed in this invention are consistent on different datasets.

[0122] Example 2

[0123] Example 2 mainly adopts the ablation experiment method to evaluate the impact of some features in the model by removing or modifying these features. Example 2 mainly discusses the impact of learnable serialization and filter encoder on the overall effect.

[0124] Fig.10 The ablation results of learnable serialization are given. Figures 4 to 6 Unlike the single-modal input used in

[11] , which does not contain multi-model information fusion, this paper introduces a learnable sequence as an additional pseudo input. In this section, MIFF-H (Houston), MIFF-T (Trento), and MIFF-M (MUUFL) represent Figures 4 to 6Similar structure. The results show that most models with additional learnable sequence outperform models without additional learnable tagging. For example, the average OA, AA, and Kappa of MIFF-M with additional sequence are 92.68%, 77.82%, and 90.33%, respectively. This shows that adding learnable sequence can improve performance.

[0125] Fig.11 Ablation results of filter encoder. The results show that filter encoder is an important component of the proposed model to improve classification performance. Specifically, the average OA, AA and Kappa of MIFF-M including filter encoder are 93.53%, 78.91% and 91.45% respectively. These results highlight the effectiveness of filter encoder in improving the overall model accuracy and robustness on different datasets.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A hyperspectral image classification method based on multimodal data, characterized in that: The following steps are involved: S1. Performing feature extraction on the hyperspectral image, and performing feature extraction on the hyperspectral image, the Lidar data corresponding to the hyperspectral image, and / or the pseudo RGB image respectively; S2, filtering the features extracted in S1 to obtain filtered information, and filtering out interference items that hinder information fusion; S3. Use a joint decoder to fuse the filtered information and predict the final classification result.

2. The hyperspectral image classification method based on multimodal data according to claim 1, characterized in that: The feature extraction described in step S1 is to extract and serialize abstract features of the hyperspectral image, the Lidar data corresponding to the hyperspectral image and / or the pseudo RGB image respectively.

3. The hyperspectral image classification method based on multimodal data according to claim 2 is characterized in that: The abstract feature extraction specifically includes the deep features and multimodal features extracted by the backbone, specifically using 2D convolution after 3D convolution and before Transformer. The process of abstract feature extraction can be defined as follows: Where: φ(I H ) represents the hyperspectral image, φ(I L ) represents the LiDAR data, φ(I R ) represents the pseudo RGB data, is the deep feature extracted by the backbone, F(I L ,I R ) is the multimodal feature, Conv2D is 2D convolution, and Conv3D is 3D convolution.

4. The hyperspectral image classification method based on multimodal data according to claim 2 is characterized in that: After the abstract feature extraction is completed, the serialization needs to be performed, and the serialization specifically includes the following steps: S11, serializing the deep features extracted from the backbone is to convert the deep features extracted from the backbone into a tokenized form, using the four different feedforward neural networks, and the specific process is as follows: X′ o =X′ f2 *X f1 X′ f2 =Softmax(X f2 ) Where: X' o is the tokenized form of the deep features extracted by the backbone, X' f2 Yes X f2 The result obtained after processing by the Softmax function, X f1 , X f2 is the feedforward neural network output of HIS, W f1 and b f1 are the learnable weights and biases of the first feedforward neural network, W f2 and b f2 are the learnable weights and biases of the second feedforward neural network, represents the deep features extracted by the backbone; S12, serializing the multimodal features is to convert the multimodal features into a tokenized form, using the four different feedforward neural networks, and the specific process is as follows: AND' o =Y′ f2 *AND f1 AND' f2 =SoftmaX(Y f2 ) Y f1 =F(I L I R )·W f3 +b f3 Y f2 =F(I L ,I R )·W f4 +b f4 Where: Y' o is the tokenized form of the multimodal feature after transformation, Y' f2 Yes f2 The result obtained after processing by the Softmax function, Y f1 , Y f2 is the feedforward neural network output of HIS, W f3 and b f3 are the learnable weights and biases of the third feedforward neural network, W f4 and b f4 are the learnable weights and biases of the fourth feedforward neural network, F(I L ,I R ) is the multimodal feature.

5. The hyperspectral image classification method based on multimodal data according to claim 1, characterized in that: The step S2 constructs the filter encoder using self-enhanced attention and filter attention to filter out interference items in each modality that hinder information fusion.

6. The hyperspectral image classification method based on multimodal data according to claim 5, characterized in that: The self-enhanced attention is used to capture the differences within a single modality. Specifically, the Transformers shared attention mechanism is used, which can distinguish the importance of the single modality information within the abstract feature. The formula is as follows: Where: MultiHeadAttn QKV is the result calculated using the attention mechanism shared by the Transformers, where Q, K, and V represent query, key, and value, respectively. k Represents the dimension of the key value.

7. The hyperspectral image classification method based on multimodal data according to claim 5, characterized in that: The filter attention is to calculate the score by selecting the abstract features from different modalities to form the query and key value to measure the relationship between different modalities. Low scores can adaptively filter out interference items, and high scores can enhance the most beneficial information for classification. Using the target and query feature sequence (X'o, Y'o), the detailed calculation in the hyperspectral image branch can be expressed as follows: Where: It's X o and Y' o The transformed dimensional feature, X' o is the tokenized form of the deep features extracted by the backbone, Y' o is the tokenized form of the multimodal feature after transformation, FFN is the front neural network, LN is the layer normalization operation, FA is the filter attention, MLP is a two-layer linear projection, is the result obtained after being processed by the pre-neural network, is the result obtained after the filtering attention processing, is a modal feature.

8. The hyperspectral image classification method based on multimodal data according to claim 1, characterized in that: The step S3 needs to enhance the information fusion capability between all modalities and perform joint decoding. The specific steps are as follows: S31. Connect all the modal features and set the first token as CLS token. This process is described as follows: Where: is the fused sequence, fusion feature The query, key and value as self-enhanced attention mechanism realizes attention fusion of all modalities, is the modal feature, It's Y' o Change the query to X' o Changing to said modal features calculated from keys and values; S32. Use multilayer perceptron as the final prediction model for classification and use cross entropy loss for classification tasks. The specific process is as follows: Where: is the final specific classification result, y i is the prediction after the self-enhanced attention and the multi-layer perceptron processing, λ i is a hyperparameter.

9. A hyperspectral image classification method based on multimodal data, characterized in that: The invention comprises at least one processor and a memory in communication with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a hyperspectral image classification method based on multimodal data according to any one of claims 1 to 8.

Images