Hyperspectral image classification method and device based on double-branch multi-scale CNN and memory-enhanced transformer

Abstract

The application discloses a hyperspectral image classification method and device based on a double-branch multi-scale CNN and a memory-enhanced Transformer. Specifically, step 1, obtain hyperspectral image data and perform preprocessing. Step 2, input the data after step 1 into a double-branch multi-scale CNN feature extraction module to fully extract the spatial features and spatial-spectral joint features of the hyperspectral image. Step 3, pass the data after step 2 into a memory module, where MT1 performs detailed memory and global memory on the data, and MT2 performs normal memory and important feature memory on the data. Step 4, pass the data after steps 2 and 3 into a memory Transformer as input. Step 5, finally, obtain a classification result through an mlp-head layer. The application can fully combine the rich spatial features and spectral features of the hyperspectral image, and improve the classification accuracy and classification performance of the hyperspectral image.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, specifically to a hyperspectral image classification method and apparatus based on a dual-branch multi-scale CNN and memory-enhanced Transformer. Background Technology

[0002] Hyperspectral remote sensing can detect a wide range of spectral bands, including ultraviolet, visible, near-infrared, and mid-infrared bands. This dense spectral band structure presents near-continuous spectral characteristics, providing more accurate and detailed descriptions of ground features, which is of great significance for remote sensing image analysis and applications. Furthermore, hyperspectral data possesses three-dimensional characteristics, meaning that different spectral bands continuously record spectral information within the same spatial scene, while preserving spatial information of ground features. This characteristic enables hyperspectral imaging to be widely used in fields such as agricultural production, biomedical imaging, geological exploration, and urban planning.

[0003] In early research on hyperspectral image classification techniques, machine learning-based classification methods, such as Support Vector Machines (SVM), Decision Trees (DT), and Random Forests (RF), typically processed the spectral features of hyperspectral images while neglecting their spatial features, resulting in less than ideal classification results. Furthermore, these methods mostly require manual feature extraction, are overly reliant on human experience, and are time-consuming, making it difficult for traditional methods to perform effectively in many situations.

[0004] With the popularity of Convolutional Neural Networks (CNNs), CNN-based hyperspectral image classification methods have emerged. Besides CNNs, several other high-performance network classes have been used for hyperspectral image classification. Transformers, originally a model for natural language processing, capture global dependencies in input sequences through self-attention mechanisms. In recent years, Transformer-based network models have also been widely applied in image processing. However, while CNNs excel at extracting local features, their receptive field is often limited to the size of the convolutional kernel, resulting in poor performance in global modeling. Transformers excel at solving long-term spectral dependencies and modeling global information, but they lack the ability to capture local context or semantic features. Summary of the Invention

[0005] The main problem this invention addresses is the complexity of hyperspectral image classification models and insufficient feature extraction; it also solves the problem of low efficiency in CNN-based and Transformer-based models.

[0006] The technical solution adopted in this invention is as follows: Firstly, this invention provides a hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer, comprising the following steps:

[0007] Step 1: Acquire hyperspectral image data and perform preprocessing;

[0008] Step 2: Input the preprocessed data from Step 1 into the dual-branch multi-scale CNN module. The dual-branch multi-scale CNN module includes 3DCNN and 2DCNN branches. The 3DCNN branch extracts both spatial and spectral features of the hyperspectral image and captures features at different scales. The 2DCNN branch extracts local spatial features and subtle differences between adjacent spectra of the hyperspectral image. The results of the two branches are fused and merged as the output.

[0009] Step 3: Input the output data from Step 2 into memory modules MT1 and MT2; MT1 performs channel-wise convolution and point-wise convolution on the output data from Step 2 to obtain P. MT1 MT2 performs average pooling and max pooling on the output data after step two and then concatenates them to obtain P. MT2 ;

[0010] Step four, combine the output data from step two with the P data from step three. MT1 and P MT2 The data is used as input to train the Memory Augmentation Transformer; the Memory Augmentation Transformer consists of two concatenated sub-layers, the first sub-layer containing a Layernorm layer and a Memory Augmentation layer (MSA); the output data from step two is processed through the Layernorm layer of the first sub-layer to obtain P. n The Memory Enhancement Layer (MSA) is used to transfer P MT1 and P MT2 After performing a linear transformation and concatenating it with the K and V of the attention mechanism, the attention mechanism is calculated, and then combined with P. n The sum is used as the output of the first sub-layer and passed to the second sub-layer for calculation. The second sub-layer contains a Layernorm layer and a feed-forward layer. The output of the feed-forward layer plus the output of the first sub-layer is used as the final output of the second sub-layer.

[0011] Step 5: Input the output of the second sub-layer in Step 4 into the mlp-head module to obtain the final classification result.

[0012] Furthermore, the preprocessing method in step one is as follows:

[0013] Step 1.1, input the original hyperspectral data with dimensions X∈R H×W×BWhere H is the height of the hyperspectral image, W is the width of the hyperspectral image, and B is the dimension of the hyperspectral image; principal component analysis is performed on the original hyperspectral data to reduce its dimension, resulting in hyperspectral data X1∈R. H×W×C C represents the dimension of the hyperspectral image after dimensionality reduction;

[0014] Step 1.2: Extract cube data patches from X1, labeled with their center pixels. The size of each patch is represented as P∈R. S×S×C , where S×S represents the width and height of the patch, which are the same size.

[0015] Furthermore, the dual-branch multi-scale CNN module in step two is specifically as follows:

[0016] The dual-branch multi-scale CNN module includes a 3DCNN branch, a 2DCNN branch, and a 2D convolution; the 3DCNN contains two 3D convolutional layers with different kernel sizes, capable of simultaneously extracting spatial and spectral features of hyperspectral images and capturing features at different scales, specifically as follows:

[0017] P 3D =Conv3D2(Conv3D1(P))

[0018] Where P represents the input data, Conv3D1 represents a 3D convolutional layer with a larger kernel, and Conv3D2 represents a 3D convolutional layer with a smaller kernel.

[0019] 2DCNN contains three 2D convolutional layers with different kernel sizes, which can be used to extract local spatial features and subtle differences between adjacent spectra in hyperspectral images, specifically as follows:

[0020] P 2D =Conv2D3(Conv2D2(Conv2D1(P)))

[0021] Where P represents the input data, and Conv2D1, Conv2D2 and Conv2D3 represent three 2D convolution operations with kernel sizes increasing sequentially.

[0022] The data obtained from the 3DCNN branch and the 2DCNN branch are fused and merged, and the result is input into the last 2D convolutional layer, specifically as follows:

[0023]

[0024] in This indicates element addition, and r1 and r2 represent the weight ratio, which are used to adjust the weight ratio between the 3D branch and the 2D branch.

[0025] Furthermore, step three is detailed as follows:

[0026] The output obtained in step two is input into memory modules MT1 and MT2 respectively. MT1 contains one channel-wise convolutional layer, one pointwise convolutional layer, and one 2D convolutional layer; MT2 contains a max-pooling layer and an average-pooling layer, specifically represented as follows:

[0027] P MT1 =Conv2D(Conv2D) p (Conv2D c (P s2 )))

[0028] Where P s2 Represented as the data after step two, Conv2D c Represented as a channel-wise convolutional layer in MT1, Conv2D p The layer is represented as a pointwise convolutional layer in MT1;

[0029] P MT2 =Concat(AvgPool(P) s2 MaxPool(P) s2 )))

[0030] Where P s2 The data after step two is represented as follows: AvgPool represents the average pooling layer in MT2, and MaxPool represents the maximum pooling layer in MT2.

[0031] Furthermore, the memory-enhancing Transformer in step four is as follows:

[0032] Step 4.1: The memory-enhancing Transformer consists of two concatenated sub-layers. The first sub-layer contains a Layernorm layer, a memory-enhancing layer (MSA), and a residual connection. The second sub-layer contains a Layernorm layer, a feed-forward layer, and a residual connection. First, the output obtained in Step 2 is embedded with positional information and concatenated with an all-zero vector as a learnable classification vector. Then, the result is fed into the Layernorm layer to obtain the data P. n The result obtained in step three is used as P. MT1 and P MT2 ;

[0033] Step 4.2, at this time [P n ,P MT1 ,P MT2 As input to the memory enhancement layer MSA, the feature vector P is first processed... n and memory vector P MT1 ,PMT2 A linear transformation is performed to obtain five self-learning matrices (W). q W k W v W m1 W m2 ),W q The matrix is ​​represented as a query matrix, W k Represented as a bond matrix, W v Represented as a value matrix, W m1 W m2 Represented as a memory matrix;

[0034] Q = P n W q

[0035] K = P n W K

[0036] V = P n W V

[0037] M1 = P MT1 W m1

[0038] M2 = P MT2 W m2

[0039] Then the MSA module concatenates K with M1 and V with M2 to reduce the loss of shallow information, enhance the communication between information, and strengthen memory.

[0040] K′=concat(K,M1)

[0041] V′=concat(V,M2)

[0042] Finally, the attention mechanism is calculated:

[0043]

[0044] Where Q represents the query, K' is the key and V' is the value, T represents the transpose operation, and d k Indicates the dimension of K';

[0045] Step 4.3, P n The data is added to the data obtained in step 4.2, first passed to the Layernorm layer, and then passed to the feed-forward layer;

[0046] F forward =Linear(Gelu(Linear(x)))

[0047] Therefore, the computation process of the overall memory enhancement Transformer is as follows:

[0048] P t1 =MSA(LN(P n ))+P n

[0049] P t2 =feed-forward(LN(P) t1 ))+P t1

[0050] Where P t1 P represents n After the first layer of the memory-enhanced Transformer, P t2 P represents t1 Data after the second layer of the memory-enhanced Transformer.

[0051] Furthermore, the mlp-head module in step five consists of a LayerNorm layer and a Linear layer:

[0052] P m =Linear(LN(P) t2 ))

[0053] Where LN represents LayerNorm, Linear represents Linear, and P represents P. m This is the final classification result.

[0054] Secondly, the present invention also provides a hyperspectral image classification device based on a two-branch multi-scale CNN and a memory-enhanced Transformer, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the hyperspectral image classification method based on a two-branch multi-scale CNN and a memory-enhanced Transformer.

[0055] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned hyperspectral image classification method based on a dual-branch multi-scale CNN and memory-enhanced Transformer.

[0056] Fourthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the hyperspectral image classification method based on a dual-branch multi-scale CNN and memory-enhanced Transformer.

[0057] The beneficial effects of this invention are as follows:

[0058] 1. This invention improves and combines the CNN structure and the Transformer structure, combining the advantages of both structures to improve the accuracy and efficiency of hyperspectral image classification;

[0059] 2. The dual-branch CNN feature extraction module in this invention employs 3DCNN and 2DCNN branches respectively, fully combining the extraction of hyperspectral spatial features and spectral-spatial joint features;

[0060] 3. The memory modules MT1 and MT2 in this invention are simple and efficient, maximizing the extraction of effective data within a minimal structure. MT1 performs detailed and global memory of the data. MT2 performs important feature memory and fuzzy memory of the data.

[0061] 4. The memory-enhanced Transformer in this invention significantly strengthens the exchange of information, reduces feature loss, and is the main structure of the model, greatly improving the model's accuracy and performance. Attached Figure Description

[0062] Figure 1 This is a flowchart of the method of the present invention;

[0063] Figure 2 This is a schematic diagram of the overall network structure of the present invention;

[0064] Figure 3 This is a schematic diagram of the dual-branch multi-scale CNN network structure of the present invention;

[0065] Figure 4 This is a schematic diagram of the memory-enhanced Transformer network structure of the present invention;

[0066] Figure 5 This refers to the classification results of the method used in the Pavia University dataset in this invention.

[0067] Figure 6 The classification results are those of the method used in the WHU-Hi-HanChuan dataset in this invention.

[0068] Figure 7 The diagram shows the structure of a hyperspectral image classification device based on a dual-branch multi-scale CNN and memory-enhanced Transformer, which is provided by the present invention. Detailed Implementation

[0069] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0070] To address the challenges of complex hyperspectral image classification models, insufficient feature extraction, and low efficiency of CNN-based and Transformer-based methods, this invention proposes a hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer. Figure 1 This is a flowchart of the present invention, and the overall network structure is as follows: Figure 2 As shown, the specific workflow is as follows:

[0071] (1) Acquire hyperspectral image data and perform preprocessing;

[0072] (1-1) The original data size of the hyperspectral image is X∈R H×W×B Where H is the height of the hyperspectral image, W is the width of the hyperspectral image, and B is the dimension of the hyperspectral image. Principal component analysis of the raw hyperspectral data can be used for dimensionality reduction, preserving the main dimensional features and removing some useless noise, to obtain the hyperspectral data X1∈R. H×W×C C represents the dimension of the hyperspectral image after dimensionality reduction.

[0073] (1-2) Extract cube data patches labeled with their center pixels from X1. The size of each patch is represented as P∈R. S×S×C Where S×S represents the width and height of the patch, which are the same size. The dataset and test set are then divided.

[0074] (2) Process the preprocessed data from step (1) and input it into the hyperspectral image feature extraction module (MCNN) of the dual-branch multi-scale CNN to perform spectral-spatial feature extraction and local spatial feature extraction on the hyperspectral data. This includes the following steps:

[0075] (2-1) The dual-branch multi-scale CNN network structure of this invention is as follows: Figure 3 As shown. The data after step (1) is input into the 3DCNN branch of MCNN. 3DCNN contains two 3D convolutional layers with different kernel sizes, which can simultaneously extract the spatial and spectral features of hyperspectral images and capture features at different scales. Specifically, it is represented as follows:

[0076] P 3D =Conv3D2(Conv3D1(P))

[0077] Where Conv3D1 represents a 3D convolutional layer with a larger kernel, and Conv3D2 represents a 3D convolutional layer with a smaller kernel.

[0078] (2-2) Input the data into the 2DCNN branch of MCNN. The 2DCNN contains three 2D convolutional layers with different kernel sizes. The kernel sizes of the three convolutional layers increase sequentially, which can be used to extract local spatial features and subtle differences between adjacent spectra in hyperspectral images. Specifically, it is represented as follows:

[0079] P 2D =Conv2D3(Conv2D2(Conv2D1(P)))

[0080] (2-3) The data after step (1) is input into the data obtained from the 3DCNN branch and the 2DCNN branch respectively, fused and merged, and the result is input into the last 2D convolutional layer. Specifically, it is expressed as follows:

[0081]

[0082] in This indicates element-wise addition. r1 and r2 represent the weight ratios, used to adjust the weight ratio between the 3D and 2D branches.

[0083] (3) Input the output obtained in step (2) into memory modules MT1 and MT2 respectively. MT1 contains one channel-wise convolutional layer, one pointwise convolutional layer, and one 2D convolutional layer. MT2 contains one max-pooling layer and one average-pooling layer. The effect of the MT1 memory module is similar to detailed memory features in real life. The max-pooling layer in MT2 is similar to important memory features in real life, and the average-pooling layer is similar to fuzzy memory features in real life. The specific structures of MT1 and MT2 are as follows:

[0084] P MT1 =Conv2D(Conv2D) p (Conv2D c (P s2 )))

[0085] Where P s2 Represented as the data after step (2), Conv2D c Represented as a channel-wise convolutional layer in MT1, Conv2D p The layer is represented as a pointwise convolutional layer in MT1.

[0086] P MT2 =Concat(AvgPool(P) s2 MaxPool(P) s2 ))

[0087] Where P s2The data after step (2) is represented as AvgPool, which represents the average pooling layer in MT2, and MaxPool, which represents the maximum pooling layer in MT2.

[0088] (4) Concatenate the data output in step (3) with K and V in the memory-enhanced Transformer. The memory-enhanced Transformer network structure is as follows: Figure 4 As shown. Step (4) specifically includes the following steps:

[0089] (4-1) The memory-enhanced Transformer consists of two concatenated sub-layers. The first sub-layer contains a Layernorm layer, a memory-enhanced MSA layer, and a residual connection. The second layer contains a Layernorm layer, a feed-forward layer, and a residual connection. First, the output obtained in step (2) is embedded with positional information and concatenated with an all-zero vector as a learnable classification vector. Then, the result is fed into the Layernorm layer to obtain data P. n The result obtained in step (3) is taken as P. MT1 and P MT2 .

[0090] (4-2) At this time [P] n ,P MT1 ,P MT2 This serves as the input to the memory enhancement layer (MSA). First, the feature vector P... n and memory vector P MT1 ,P MT2 A linear transformation is performed to obtain five self-learning matrices (W). q W k W v W m1 W m2 ),W q The matrix is ​​represented as a query matrix, W k Represented as a bond matrix, W v Represented as a value matrix, W m1 W m2 It is represented as a memory matrix.

[0091] Q = P n W q

[0092] K = P n W K

[0093] V = P n W V

[0094] M1 = P MT1 Wm1

[0095] M2 = P MT2 W m2

[0096] Then, in the memory enhancement layer MSA, K is concatenated with M1, and V with M2. This reduces the loss of shallow information, strengthens information exchange between shallow and deep structures, and enhances information retention.

[0097] K′=concat(K,M1)

[0098] V′=concat(V,M2)

[0099] Finally, the attention mechanism is calculated:

[0100]

[0101] Where Q represents the query, K' is the key and V' is the value, T represents the transpose operation, and d k This indicates the dimension of K'.

[0102] (4-3) P n The data obtained from (4-2) is added together and first passed to the Layernorm layer, and then passed to the feed-forward layer. The feed-forward layer calculation process is as follows:

[0103] F forward =Linear(Gelu(Linear(x)))

[0104] Therefore, the computation process of the overall memory enhancement Transformer is as follows:

[0105] P t1 =MSA(LN(P n ))+P n

[0106] P t2 =feed-forward(LN(P) t1 ))+P t1

[0107] Where P t1 P represents n Data after the first layer of the memory-enhanced Transformer. P t2 P represents t1 Data after the second layer of the memory-enhanced Transformer.

[0108] (5) Transfer the data processed by the memory-enhanced Transformer to the mlp-head layer for final result classification.

[0109] P m =Linear(LN(P) t2 ))

[0110] Where LN represents Layernorm, Linear represents Linear, and P represents P. m This indicates the final classification result.

[0111] The effectiveness of the method of the present invention is further confirmed by the following experiments:

[0112] Method evaluation includes overall accuracy (OA), average accuracy (AA), and Kappa coefficient (KAPPA).

[0113] Overall accuracy (OA) represents the proportion of correctly classified samples out of the total number of classified samples;

[0114] Average precision (AA) represents the average classification accuracy for each category;

[0115] The Kappa coefficient (KAPPA) is used to measure the consistency of a classification model, taking into account the difference between the proportion of correct predictions made by the model and the accuracy of simply guessing randomly.

[0116] The patch size is 13*13, the training samples from Pavia University are 0.5%, and the training samples from WHU-Hi-HanChuan are 1%.

[0117] The classification results for the Pavia University and WHU-Hi-HanChuan datasets are as follows: Figure 5 , Figure 6 As shown.

[0118] Table 1. Comparison of accuracy of Pavia University classification methods

[0119]

[0120]

[0121] Table 2 Comparison of accuracy of various classification methods in WHU-Hi-HanChuan

[0122]

[0123]

[0124] As shown in Tables 1 and 2, the accuracy of this invention across multiple categories on the Pavia University and WHU-Hi-HanChuan datasets is higher than that of many other existing methods. Furthermore, the accuracy for OA, AA, and KAPPA is also higher than many other existing classification methods. In Pavia University, compared to HyBridSN, RSSAN, SSFTT, and morphFormer, the accuracy of this invention in OA is 5.51%, 4.09%, 2.43%, and 2.98%, respectively; in AA, it is 9.02%, 6.95%, 3.14%, and 4.57%, respectively; and in KAPPA, it is 7.37%, 5.49%, 3.25%, and 3.95%, respectively. Furthermore, as shown in Table 1, the Transformer-based network architecture model has better accuracy than the CNN-based network architecture model on Pavia University. In the WHU-Hi-HanChuan dataset, compared to HyBridSN, RSSAN, SSFTT, and morphFormer, this invention achieves higher OA (Objective Analytical Capabilities) by 1.4%, 4.18%, 1.49%, and 1.16%, respectively; higher AA (Accuracy Analytical Capabilities) by 4.87%, 8.43%, 2.82%, and 2.16%, respectively; and higher KAPPA (Accuracy Analytical Capabilities) by 1.64%, 4.89%, 1.74%, and 1.36%, respectively. Table 2 shows that in the WHU-Hi-HanChuan dataset, HyBridSN achieves higher accuracy in the first few classes than Transformer-based network architecture models. However, the overall accuracy of CNN-based network architecture models is still lower than that of Transformer-based network architecture models.

[0125] Corresponding to the aforementioned embodiment of a hyperspectral image classification method based on a dual-branch multi-scale CNN and memory-enhanced Transformer, the present invention also provides an embodiment of a hyperspectral image classification device based on a dual-branch multi-scale CNN and memory-enhanced Transformer.

[0126] See Figure 7 The present invention provides a hyperspectral image classification device based on a two-branch multi-scale CNN and a memory-enhanced Transformer, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a hyperspectral image classification method based on a two-branch multi-scale CNN and a memory-enhanced Transformer as described in the above embodiment.

[0127] The present invention provides an embodiment of a hyperspectral image classification device based on a dual-branch multi-scale CNN and a memory-enhanced Transformer. This embodiment can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which includes a hyperspectral image classification device based on a dual-branch multi-scale CNN and memory-enhanced Transformer provided by the present invention. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0128] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0129] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0130] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a hyperspectral image classification method based on a dual-branch multi-scale CNN and memory-enhanced Transformer as described in the above embodiments.

[0131] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0132] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned hyperspectral image classification method based on a dual-branch multi-scale CNN and memory-enhanced Transformer.

[0133] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer, characterized in that, Includes the following steps: Step 1: Acquire hyperspectral image data and perform preprocessing; Step 2: Input the preprocessed data from Step 1 into the dual-branch multi-scale CNN module. The dual-branch multi-scale CNN module includes 3DCNN and 2DCNN branches. The 3DCNN branch extracts both spatial and spectral features of the hyperspectral image and captures features at different scales. The 2DCNN branch extracts local spatial features of the hyperspectral image and subtle differences between adjacent spectra. The results of the two branches are fused and merged as the output. Step 3: Input the output data from Step 2 into memory modules MT1 and MT2; MT1 performs channel-wise convolution and point-wise convolution on the output data after step two to obtain P. MT1 MT2 performs average pooling and max pooling on the output data after step two and then concatenates them to obtain P. MT2 ; Step four, combine the output data from step two with the P data from step three. MT1 and P MT2 The data is used as input to train the Memory Augmentation Transformer; the Memory Augmentation Transformer consists of two concatenated sub-layers, the first sub-layer containing a Layernorm layer and a Memory Augmentation layer (MSA); the output data from step two is processed through the Layernorm layer of the first sub-layer to obtain P. n The Memory Enhancement Layer (MSA) is used to transfer P MT1 and P MT2 After performing a linear transformation and concatenating it with the K and V of the attention mechanism, the attention mechanism is calculated, and then combined with P. n The sum is used as the output of the first sub-layer and passed to the second sub-layer for calculation. The second sub-layer contains a Layernorm layer and a feed-forward layer. The output of the feed-forward layer plus the output of the first sub-layer is used as the final output of the second sub-layer. Step 5: Input the output of the second sub-layer in Step 4 into the mlp-head module to obtain the final classification result.

2. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 1, characterized in that, The preprocessing method in step one is as follows: Step 1.1, input the original hyperspectral data with dimensions X∈R H×W×B Where H is the height of the hyperspectral image, W is the width of the hyperspectral image, and B is the dimension of the hyperspectral image; principal component analysis is performed on the original hyperspectral data to reduce its dimension, resulting in hyperspectral data X1∈R. H×W×C C represents the dimension of the hyperspectral image after dimensionality reduction; Step 1.2: Extract cube data patches from X1, labeled with their center pixels. The size of each patch is represented as P∈R. S×S×C , where S×S represents the width and height of the patch, which are the same size.

3. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 1, characterized in that, The specific details of the dual-branch multi-scale CNN module in step two are as follows: The dual-branch multi-scale CNN module includes a 3DCNN branch, a 2DCNN branch, and a 2D convolution; the 3DCNN contains two 3D convolutional layers with different kernel sizes, capable of simultaneously extracting spatial and spectral features of hyperspectral images and capturing features at different scales, specifically as follows: P 3D =Conv3D2(Conv3D1(P)) Where P represents the input data, Conv3D1 represents a 3D convolutional layer with a larger kernel, and Conv3D2 represents a 3D convolutional layer with a smaller kernel. 2DCNN contains three 2D convolutional layers with different kernel sizes, which can be used to extract local spatial features and subtle differences between adjacent spectra in hyperspectral images, specifically as follows: P 2D =Conv2D3(Conv2D2(Conv2D1(P))) Where P represents the input data, and Conv2D1, Conv2D2 and Conv2D3 represent three 2D convolution operations with kernel sizes increasing sequentially. The data obtained from the 3DCNN branch and the 2DCNN branch are fused and merged, and the result is input into the last 2D convolutional layer, specifically as follows: in This indicates element addition, and r1 and r2 represent the weight ratio, which are used to adjust the weight ratio between the 3D branch and the 2D branch.

4. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 1, characterized in that, Step three is detailed as follows: The output obtained in step two is input into memory modules MT1 and MT2 respectively. Module MT1 contains one channel-wise convolutional layer, one pointwise convolutional layer, and one 2D convolutional layer; module MT2 contains a max pooling layer and an average pooling layer, specifically represented as follows: P MT1 =Conv2D(Conv2D p (Conv2D c (P s2 ))) Where P s2 Represented as the data after step two, Conv2D c Represented as a channel-wise convolutional layer in MT1, Conv2D p The layer is represented as a pointwise convolutional layer in MT1; P MT2 =Concat(AvgPool(P s2 )),MaxPool(P s2 ))) Where P s2 The data after step two is represented as follows: AvgPool represents the average pooling layer in MT2, and MaxPool represents the maximum pooling layer in MT2.

5. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 1, characterized in that, The memory enhancement Transformer in step four is as follows: Step 4.1: The memory-enhancing Transformer consists of two concatenated sub-layers. The first sub-layer contains a Layernorm layer, a memory-enhancing layer (MSA), and a residual connection. The second sub-layer contains a Layernorm layer, a feed-forward layer, and a residual connection. First, the output obtained in Step 2 is embedded with positional information and concatenated with an all-zero vector as a learnable classification vector. Then, the result is fed into the Layernorm layer to obtain the data P. n The result obtained in step three is used as P. MT1 and P MT2 ; Step 4.2, at this time [P n ,P MT1 ,P MT2 As input to the memory enhancement layer MSA, the feature vector P is first processed... n and memory vector P MT1 ,P MT2 A linear transformation is performed to obtain five self-learning matrices (W). q W k W v W m1 W m2 ),W q The matrix is ​​represented as a query matrix, W k Represented as a bond matrix, W v Represented as a value matrix, W m1 W m2 Represented as a memory matrix; Q=P n W q K=P n W K V=P n W V M1=P MT1 W m1 M2=P MT2 W m2 Then the MSA module concatenates K with M1 and V with M2 to reduce the loss of shallow information, enhance the communication between information, and strengthen memory. K′=concat(K,M1) V′=concat(V,M2) Finally, the attention mechanism is calculated: Where Q represents the query, K' is the key and V' is the value, T represents the transpose operation, and d k Indicates the dimension of K'; Step 4.3, P n The data is added to the data obtained in step 4.2, first passed to the Layernorm layer, and then passed to the feed-forward layer; F forward =Linear(Gelu(Linear(x))) Therefore, the computation process of the overall memory enhancement Transformer is as follows: P t1 =MSA(LN(P n ))+P n P t2 =feed-forward(LN(P t1 ))+P t1 Where P t1 P represents n After the first layer of the memory-enhanced Transformer, P t2 P represents t1 Data after the second layer of the memory-enhanced Transformer.

6. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 5, characterized in that, The mlp-head module in step five consists of a LayerNorm layer and a Linear layer: P m =Linear(LN(P t2 )) Where LN represents LayerNorm, Linear represents Linear, and P represents P. m This is the final classification result.

7. A hyperspectral image classification device based on a dual-branch multi-scale CNN and a memory-enhanced Transformer, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a hyperspectral image classification method based on a dual-branch multi-scale CNN and memory-enhanced Transformer as described in any one of claims 1-6.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a hyperspectral image classification method based on a dual-branch multi-scale CNN and memory-enhanced Transformer as described in any one of claims 1-6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a hyperspectral image classification method based on a dual-branch multi-scale CNN and memory-enhanced Transformer as described in any one of claims 1-6.

Images