Multi-modal hyperspectral image classification method and system

Abstract

The invention discloses a multi-modal hyperspectral image classification method and system, and the method comprises the steps: extracting the spatial features and pixel-level features of a hyperspectral image and other modal images based on a 3D cube and an original center pixel, fully mining the spatial spectral features and unique features of HSI and other modal images based on an image block and the center pixel, and carrying out the classification of the hyperspectral image and other modal images. According to the method, the multi-modal tokens are input into a feature matrix, the single-modal tokens and the mixed-modal tokens are constructed on the basis, the multi-modal features are adaptively fused at the calculation cost linearly related to the line number of the input feature matrix, information loss is reduced, the classification effect is improved, meanwhile, the generated tokens not only comprise the single-modal tokens, but also comprise the multi-modal fusion tokens, and the classification efficiency is improved. Various feature fusion conditions are comprehensively considered, and information loss caused in the feature fusion process is reduced as much as possible.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and relates to a multimodal hyperspectral image classification method and system. Background Art

[0002] In response to global environmental challenges such as climate change and desertification, detailed observation of the Earth has become essential. As these demands continue to grow, remote sensing technology has become an important tool for scientific research and environmental monitoring with its unique advantages. In recent years, with the development of remote sensing technology, remote sensing image classification has gradually become one of the hot topics in the field of remote sensing image research. The goal of remote sensing image classification is to assign land cover categories to each pixel in the remote sensing image, where each category represents a specific land feature, such as water, vegetation, urban buildings, etc. Remote sensing image classification has been widely used in many studies such as forestry, mineral exploration and mapping, target detection, environmental monitoring, urban planning, biodiversity conservation, and disaster response and management.

[0003] In order to further improve the accuracy of remote sensing image classification, data from different sensors are introduced for classification based on hyperspectral image classification, which is called multimodal remote sensing image classification. Specifically, for the same geographical area, different sensors are used to capture data, and the obtained data can provide information with different characteristics for the same observation surface coverage area. Each modality can provide unique information complementary to HSI: LiDAR collects depth and intensity information and measures the height of objects on the earth's surface, SAR data provides amplitude and phase geometric information, and DSM can provide the three-dimensional morphology of the earth's surface. Multimodal remote sensing image classification through deep learning requires extracting deep learning features from HSI and other modal images, and then inputting the features into the classifier for prediction. The main challenges of this task are: 1) Remote sensing images of other modalities such as LiDAR, SAR and DSM each contain unique information. How to extract reliable information from them and use it for subsequent information fusion is a major problem. 2) Due to its unique network architecture, Transformer can flexibly integrate different types of data, which makes it very suitable for performing multimodal tasks. How to use this feature to assist methods in fusing multimodal remote sensing image features while ensuring that too much information is not lost is a question worth considering.

[0004] Traditional methods fuse HSI with LiDAR morphological features (such as morphological features and attribute features) through subspace learning and other methods to enhance the classification effect. In addition, global fusion techniques such as extinction features (EPs) are also used to improve the ability to extract multimodal features. Traditional classifiers such as support vector machines (SVMs) show good potential in the combination of multiple features and multi-source data environments. The emergence of deep learning models has greatly improved the ability to classify multimodal remote sensing data, especially the performance of convolutional neural networks (CNNs) in feature extraction and fusion. Through structures such as 1-D, 2-D and 3-D CNNs, spatial spectral features can be effectively extracted from HSI, and elevation information can be obtained from LiDAR data. Multimodal fusion at the feature level and decision level can be performed using strategies such as parameter sharing to further optimize classification performance.

[0005] In recent years, the Transformer model has gradually become the core technology in multimodal remote sensing image classification due to its long-distance dependency modeling ability. The Transformer network processes HSI and LiDAR data through the multi-head self-attention mechanism (MSA) to achieve feature extraction and deep fusion. In order to improve the complementarity and compatibility between different modalities, the study uses modules such as multi-scale self-attention and cross-attention to fuse modal features. In addition, combined with the modal comprehensive features generated by the modal attention module, the Transformer encoder further improves the fusion effect between HSI and other modal data. In the existing technology, when classifying multimodal hyperspectral images, although the introduction of data from different sensors can provide richer feature information for the same geographical area, each modal data (such as HSI, LiDAR, SAR, DSM) contains unique and complementary information, and its information structure is complex, making it difficult to fully extract effective information from it for subsequent information fusion; secondly, how to effectively extract and fuse spatial, spectral and geometric features in multimodal data is still a difficult problem in classification tasks. In addition, although Transformers have the advantage of flexibly integrating multimodal data, their network architecture is complex and computationally intensive, which hinders their deployment in practical applications. Summary of the invention

[0006] The purpose of the present invention is to solve the problems in the prior art that the information structure of each modal data is complex, it is difficult to fully extract effective information from it for subsequent information fusion, it is impossible to effectively extract and fuse spatial, spectral and geometric features, and the network architecture is complex and the amount of calculation is large, which hinders their deployment in practical applications. A multimodal hyperspectral image classification method and system are provided.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A multimodal hyperspectral image classification method comprises the following steps:

[0009] Taking the pixel as the center, extracting the 3D cube and the original center pixel of the original hyperspectral image data, and extracting the 3D cube and the original center pixel of the image data of other modalities;

[0010] Extract spatial and pixel-level features of hyperspectral images and other modality images based on 3D cubes and original center pixels;

[0011] The spatial features and pixel-level features are converted into a token set including a single modality and a fusion modality, and the token set is fused;

[0012] Classify the fused token set to obtain the classification result.

[0013] A further improvement of the present invention is:

[0014] The method of extracting the original hyperspectral image data and other modality image data into a pixel-centered 3D cube includes:

[0015]

[0016] The extracting of the original central pixels of the original hyperspectral image data and other modality image data comprises:

[0017]

[0018] Where, PT M1 Represents a hyperspectral image patch; PT M2 Represents other modality image blocks; PX M1 The central pixel of the hyperspectral image; PX M2 The center pixel of other modality images; M1 represents HSI, M2 represents other modality remote sensing data, P is the width and height of the image data, and C is the number of channels of the hyperspectral image.

[0019] Based on PT M1 and PT M2 The spatial spectral features of HSI and the spatial features of other modality images are extracted through the image block convolution module;

[0020] Based on PX M1 and PX M2 The detailed spectral features of HSI and the unique pixel-level features of other modality images are extracted through the pixel convolution module.

[0021] The method of extracting spatial features of hyperspectral images and other modality images comprises the following steps:

[0022]

[0023] Where: PT M1 Represents a hyperspectral image patch; PT M2 Represents other modality image blocks; Reshape(·) represents a reshaping operation; Conv2D(·) represents a two-dimensional convolution layer, a batch normalization layer, and a ReLU activation layer; Conv3D(·) represents a three-dimensional convolution layer, a batch normalization layer, and a ReLU activation layer.

[0024] The method of extracting pixel-level features of hyperspectral images and other modality images comprises the following steps:

[0025]

[0026] Where: PX Mi Represents the central pixel, where Mi represents the hyperspectral image M1 or other modality image M2; Linear(·) represents a linear layer with an output dimension of d; Conv1D(·) represents a one-dimensional convolutional layer, a batch normalization layer, and a ReLU activation layer.

[0027] The conversion of spatial features and pixel-level features into a token set including a single modality and a fusion modality includes:

[0028] Hyperspectral images and single modality tokens T generated by combining hyperspectral images 11 ;

[0029] The fused modality token T generated by combining hyperspectral images with other modality images 12 ;

[0030] The fused modality token T generated by combining other modality images with the hyperspectral image 21 ;

[0031] The other modality images are combined with other modality images to generate a single modality token T 22 ;

[0032] The single modal tokens and multimodal fusion tokens are further fused through the Transformer hybrid feature module, and the results are layer-normalized and globally average pooled to obtain the fusion result.

[0033] Use a multi-layer perceptron to classify the fused token set.

[0034] A multimodal hyperspectral image classification system, comprising:

[0035] A data processing module, used for extracting a 3D cube and an original center pixel of original hyperspectral image data with a pixel as the center, and extracting a 3D cube and an original center pixel of image data of other modalities;

[0036] Feature extraction module, used to extract spatial features and pixel-level features of hyperspectral images and other modality images based on 3D cubes and original central pixels;

[0037] A fusion module is used to convert spatial features and pixel-level features into a token set containing a single modality and a fusion modality, and fuse the token set;

[0038] The classification module is used to classify the fused token set and obtain the classification result.

[0039] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any method described in the present invention when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any method described in the present invention.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention discloses a multimodal hyperspectral image classification method. The present invention extracts spatial features and pixel-level features of hyperspectral images and other modal images based on 3D cubes and original central pixels. Based on image blocks and central pixels, the spatial spectral features and unique features of HSI and other modal images are fully mined. Single modal tokens and mixed modal tokens are constructed on this basis. Multimodal features are adaptively fused at a computational cost linearly related to the number of rows of an input feature matrix, thereby reducing information loss and improving classification effect. At the same time, the generated tokens include not only single modal tokens but also multimodal fusion tokens, comprehensively considering various feature fusion situations and minimizing information loss caused by the feature fusion process as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 It is the overall structure diagram of the model of the present invention;

[0045] Figure 2 This is a structural diagram of the residual cross-attention tokenization module of the present invention;

[0046] Figure 3 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0048] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0050] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0051] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0052] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0053] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0054] See also Figures 1 to 3 The embodiment of the present invention discloses a multimodal hyperspectral image classification method, which fully exploits the features of HSI and other modal images, and at the same time, better fuses the features, reduces the loss of information, and achieves a good effect of hyperspectral image classification.

[0055] This method introduces four effective modules, including image block convolution module, pixel convolution module, residual cross attention tokenization module and Transformer hybrid feature module, where:

[0056] The image patch convolution module focuses on image patches and extracts the spatial spectral features of hyperspectral images and the spatial features of other modality images.

[0057] The pixel convolution module focuses on the central pixel and extracts the detailed spectral features of the hyperspectral image and the unique pixel-level features of other modality images.

[0058] The residual cross-attention tokenization module is used to fuse the above features and convert them into single modality tokens and fused modality tokens, which not only ensures the fusion of features, but also retains the separate features of each modality and avoids information loss.

[0059] The Transformer hybrid feature module is used to establish long-range dependencies between tokens.

[0060] The above four modules are applied sequentially to predict the label of each pixel to further improve the classification performance.

[0061] The specific steps include:

[0062] Step 1: Preprocess the input hyperspectral image and other modality images, extract the pixel-centered 3D cube and the original center pixel, and obtain the processed training set and test set data, including:

[0063] For the input hyperspectral image and other modal images Normalize each channel and crop the image blocks corresponding to the pixels to be classified from the images of the two modalities:

[0064]

[0065] Original center pixel:

[0066]

[0067] Where M1 represents HSI, M2 represents other modal remote sensing data, P is the width and height of the image data, and C is the number of channels of the hyperspectral image.

[0068] Furthermore, other modality data include LiDAR, SAR or DSM image data.

[0069] Step 2: The output of the preprocessing module is passed into the image block convolution module and the pixel convolution module to extract the spatial features and pixel-level features of the hyperspectral image and other modal images, including:

[0070] The image block convolution module uses different CNNs to extract the spatial spectral features of HSI (hyperspectral image) and the spatial features of other modality images.

[0071] The goal of the pixel convolution module is to focus on mining pixel-level features of multimodal remote sensing images.

[0072] Specifically, the network diagram of the image block convolution module is as follows: Figure 1 As shown, for HSI block PT M1 , first reshape it into 1×C M1 ×p×p size, and then input it into a three-dimensional convolution layer with 8 output channels, a convolution kernel size of 3×3×3, a step size of 1, and a pooling size of 0×1×1. In this way, the output features of size 8×(CM1-2)×p×p are obtained. Then, it is reshaped, and the first dimension and the second dimension are merged to obtain a feature matrix of size (8CM1-16)×p×p. It is input into a two-dimensional convolution with an output channel of d, a convolution kernel size of 3×3, a step size of 1, and a pooling size of 0, and a feature matrix of size d×p′×p′ is obtained, where p′=p-2. The matrix is ​​further reshaped to (p′×p′)×d, which is the spatial spectral feature corresponding to the HSI block. For other modality image blocks PT M2 , which is first reshaped into C M2 ×p×p cube, and then input into a two-dimensional convolution with an output channel of d, a convolution kernel size of 3×3, a stride of 1, and a pooling size of 0, to obtain a feature matrix of size d×p′×p′, and finally reshaped into a feature matrix of (p′×p′)×d Represents spatial features of other modalities.

[0073] The above process can be expressed as:

[0074]

[0075] Where: Reshape (·) is a reshaping operation; Conv2D (·) contains a two-dimensional convolution layer, a batch normalization layer, and a ReLU activation layer; Conv3D (·) contains a three-dimensional convolution layer, a batch normalization layer, and a ReLU activation layer. Finally, the two image block-level features, namely the spatial spectral features of HSI and spatial features of images from other modalities This is the output of this module.

[0076] Furthermore, the pixel convolution module network diagram is as follows: Figure 1 As shown. The central pixel of HSI and other modality images and As the input of this module, it is first reshaped into a size of 1×C M1 and 1×C M2 Then, it is input into a one-dimensional convolution layer with an output channel of 4, a convolution kernel size of 3, a stride of 1, and a pooling size of 1, and convolution is performed on the channel dimension to obtain 4×C M1 and 4×C M2 The features are then input into the batch normalization layer and the ReLU activation layer. Next, the features are concatenated in the second dimension and reshaped into a size of 1×4C M1 and 1×4C M2 Finally, the two feature vectors are input into the linear layer with output dimension d to get the output and The above process can be expressed as:

[0077]

[0078] Where: Mi is M1 or M2; Linear(·) is a linear layer with an output dimension of d; Conv1D(·) contains a one-dimensional convolutional layer, a batch normalization layer, and a ReLU activation layer. Finally, the detailed spectral features of HSI Unique pixel-level features from other modality images This is the output of this module.

[0079] Step 3: Based on spatial features and pixel-level features, the multimodal features are converted into tokens containing a single modality and a fusion modality using the residual cross-attention tokenization module. The Transformer hybrid feature module is used to further establish long-distance dependencies on the token set containing multimodal heterogeneous features, including:

[0080] Specifically, the multimodal features are converted into tokens containing a single modality and a fusion modality by using the residual cross-attention tokenization module. The Transformer hybrid feature module is used to further establish long-distance dependencies on the token set containing multimodal heterogeneous features.

[0081] Furthermore, the residual cross attention tokenization module is as follows Figure 2 As shown. It is used to further transform the heterogeneous features of each modality into a set of tokens for input into Transformer for classification. The residual cross-attention tokenization module receives the spatial spectral features of HSI extracted on the image block. Spatial features of images from other modalities Detailed spectral features of HSI Unique pixel-level features of images from other modalities The corresponding key matrix and value matrix are generated by mapping the weight matrix, and the residual vector is obtained by global average pooling. and It is mapped into a one-dimensional query matrix through the weight matrix, and then each query matrix is ​​combined with each pair of key matrix and value matrix one-to-one to calculate the cross attention. Finally, the residual vector is added to form a residual connection to enhance the classification effect, and a series of tokens T = [T 11 ,T 12 ,T 21 ,T 22 ] as the output of this module. The process can be summarized as:

[0082]

[0083] Where: 1D-RCA(·)——one-dimensional residual cross attention, the calculation method is as follows:

[0084]

[0085] Where: Mj and Mk——M1 (HSI) or M2 (other modality images); The i-th attention head and i∈[1,h]; ——Query matrix; ——key matrix; ——value matrix; and ——Learnable weight matrix, all attention heads are finally concatenated in the second dimension to obtain an output of size 1×d (d h ×h=d). Residual vector R Mj It is used to overcome the residual connection added by the gradient explosion, and its generation method is as follows:

[0086]

[0087] in By It is obtained by performing global average pooling (GAP) on the first dimension.

[0088] Since in 1D-RCA, the query matrix is a one-dimensional vector, so 1D-RCA has lower computational complexity than ordinary MSA:

[0089] Ω(MSA)=2p′ 4 d+4p′ 2 d 2

[0090] Ω(1D-RCA)=p′ 2 d 2 +3p′ 2 d+d 2 +d

[0091] Where: p′ 2 ——Feature Matrix and It can be seen that the global MSA is quadratically related to the number of rows of the input feature matrix, while the 1D-RCA is linearly related to the number of rows of the input feature matrix. Therefore, 1D-RCA is more suitable than MSA for processing input feature matrices with a large number of rows.

[0092] Furthermore, the Transformer hybrid feature module is as follows Figure 1 As shown in the figure, the function is to perform one-to-one information fusion of the input token set, establish long-distance dependencies, and better handle serialized features. Use two Transformer encoders to receive the token set T to globally establish long-distance dependencies between the input token sets, and further fuse the single modal tokens and multimodal fusion tokens. Finally, the result is layer normalized and globally averaged pooled to obtain T gap , as the final output of this network.

[0093] In the present invention, the residual cross-attention tokenization module is used to further convert the heterogeneous features of each modality into a set of tokens for input into Transformer for classification. The generated tokens include not only the tokens of each single modality, but also the multimodal fusion tokens, which comprehensively considers various feature fusion situations and minimizes the information loss caused by the feature fusion process. At the same time, the residual connection is introduced to further retain the original spatial spectral features. In terms of computational complexity, 1D-RCA is combined with PXConv to make the query matrix a one-dimensional vector, which reduces the computational complexity of 1D-RCA itself, and the module reconstructs the spatial spectral features into only 4 tokens, which greatly reduces the computational burden in subsequent modules.

[0094] Furthermore, in the present invention, the Transformer hybrid feature module is composed of multiple connected Transformer encoders, a normalization layer and a global average pooling layer, which can perform one-to-one information fusion of the input token set, establish long-distance dependencies, and better handle serialized features. In order to minimize the loss during multimodal feature fusion, the Transformer hybrid feature module globally re-integrates the four tokens containing different modal combinations generated by the residual cross-attention tokenization module, and uses adaptive extraction of decisive features that are conducive to classification to establish long-distance dependencies between tokens.

[0095] Step 4: Input the embedded features into the multi-layer perceptron to classify the test samples and obtain the classification results.

[0096] Assuming there are N categories, the output of the Transformer mixed feature module Input into the multilayer perceptron to obtain the classification result The commonly used cross entropy loss is used to train the proposed method. During the training phase, the cross entropy loss between Y and the true label is calculated to update the network.

[0097] This embodiment fully exploits the spatial spectral characteristics and unique features of HSI and other modal images based on image blocks and central pixels respectively, and constructs single modal tokens and mixed modal tokens, adaptively fusing multimodal features at a computational cost linearly related to the number of rows of the input feature matrix, thereby reducing information loss and improving classification results.

[0098] A multimodal hyperspectral image classification system, characterized by comprising:

[0099] A data processing module, used for extracting a 3D cube and an original center pixel of original hyperspectral image data with a pixel as the center, and extracting a 3D cube and an original center pixel of image data of other modalities;

[0100] Feature extraction module, used to extract spatial features and pixel-level features of hyperspectral images and other modality images based on 3D cubes and original central pixels;

[0101] A fusion module is used to convert spatial features and pixel-level features into a token set containing a single modality and a fusion modality, and fuse the token set;

[0102] The classification module is used to classify the fused token set and obtain the classification results

[0103] A schematic diagram of a terminal device provided in an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0104] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0105] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0106] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0107] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0108] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0109] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal hyperspectral image classification method, characterized in that: The following steps are involved: Taking the pixel as the center, extracting the 3D cube and the original center pixel of the original hyperspectral image data, and extracting the 3D cube and the original center pixel of the image data of other modalities; Extract spatial and pixel-level features of hyperspectral images and other modality images based on 3D cubes and original central pixels; The spatial features and pixel-level features are converted into a token set including a single modality and a fusion modality, and the token set is fused; Classify the fused token set to obtain the classification result.

2. A multimodal hyperspectral image classification method according to claim 1, characterized in that: The method of extracting the original hyperspectral image data and other modality image data into a pixel-centered 3D cube includes: The extracting of the original central pixels of the original hyperspectral image data and other modality image data comprises: Where, PT M1 Represents a hyperspectral image patch; PT M2 Represents other modality image blocks; PX M1 The central pixel of the hyperspectral image; PX M2 The center pixel of other modality images; M1 represents HSI, M2 represents other modality remote sensing data, P is the width and height of the image data, and C is the number of channels of the hyperspectral image.

3. A multimodal hyperspectral image classification method according to claim 2, characterized in that: Based on PT M1 and PT M2 The spatial spectral features of HSI and the spatial features of other modality images are extracted through the image block convolution module; Based on PX M1 and PX M2 The detailed spectral features of HSI and the unique pixel-level features of other modality images are extracted through the pixel convolution module.

4. The multimodal hyperspectral image classification method according to claim 1, characterized in that: The method of extracting spatial features of hyperspectral images and other modality images comprises the following steps: Where: PT M1 Represents a hyperspectral image patch; PT M2 Represents other modality image blocks; Reshape(·) represents a reshaping operation; Conv2D(·) represents a two-dimensional convolution layer, a batch normalization layer, and a ReLU activation layer; Conv3D(·) represents a three-dimensional convolution layer, a batch normalization layer, and a ReLU activation layer.

5. The multimodal hyperspectral image classification method according to claim 1, characterized in that: The method of extracting pixel-level features of hyperspectral images and other modality images comprises the following steps: Where: PX Mi Represents the central pixel, where Mi represents the hyperspectral image M1 or other modality image M2; Linear(·) represents a linear layer with an output dimension of d; Conv1D(·) represents a one-dimensional convolutional layer, a batch normalization layer, and a ReLU activation layer.

6. The multimodal hyperspectral image classification method according to claim 1, characterized in that: The conversion of spatial features and pixel-level features into a token set including a single modality and a fusion modality includes: Hyperspectral images and single modality tokens T generated by combining hyperspectral images 11 ; The fused modality token T generated by combining hyperspectral images with other modality images 12 ; The fused modality token T generated by combining other modality images with the hyperspectral image 21 ; The other modality images are combined with other modality images to generate a single modality token T 22 ; The single modal tokens and multimodal fusion tokens are further fused through the Transformer hybrid feature module, and the results are layer-normalized and globally average pooled to obtain the fusion result.

7. The multimodal hyperspectral image classification method according to claim 1, characterized in that: During classification, a multi-layer perceptron is used to classify the fused token set.

8. A multimodal hyperspectral image classification system, characterized in that: include: A data processing module, used for extracting a 3D cube and an original center pixel of original hyperspectral image data with a pixel as the center, and extracting a 3D cube and an original center pixel of image data of other modalities; Feature extraction module, used to extract spatial features and pixel-level features of hyperspectral images and other modality images based on 3D cubes and original central pixels; A fusion module is used to convert spatial features and pixel-level features into a token set containing a single modality and a fusion modality, and fuse the token set; The classification module is used to classify the fused token set and obtain the classification result.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images