Hyperspectral image classification method and device based on double-branch multi-scale CNN (Convolutional Neural Network) and memory enhancement Transform

Abstract

The invention discloses a hyperspectral image classification method and a hyperspectral image classification device based on a double-branch multi-scale CNN (Convolutional Neural Network) and a memory enhancement Transform. The method specifically comprises the following steps: step 1, acquiring and preprocessing hyperspectral image data; and step 2, inputting the data after the step 1 into a double-branch multi-scale CNN feature extraction module, and fully extracting spatial features and spatial-spectrum combined features of the hyperspectrum. And step 3, transmitting the data after the step 2 into a memory module, performing detail memory and global memory on the data by MT1, and performing normal memory and important feature memory on the data by MT2. 4, the data obtained in the step 2 and the data obtained in the step 3 serve as input and are transmitted into a memory Transform; and 5, finally, a classification result is obtained through an mlp-head layer. According to the method, rich spatial features and spectral features of the hyperspectrum can be fully combined, and the classification precision and the classification performance of the hyperspectral image are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a hyperspectral image classification method and device based on a dual-branch multi-scale CNN and a memory-enhanced Transformer. Background Art

[0002] Hyperspectral remote sensing can perceive a wide range of spectral bands, including ultraviolet, visible light, near infrared and mid-infrared bands. This dense spectral band presents nearly continuous spectral characteristics, providing a more accurate and detailed description of ground object information, which is of great significance for remote sensing image analysis and application. In addition, hyperspectral data has the characteristics of three-dimensionality, that is, different spectral bands continuously record spectral information in the same spatial scene, while retaining the spatial information of the ground object surface. This feature makes hyperspectral imaging widely used in agricultural production, biomedical imaging, geological exploration and urban planning.

[0003] In the early research on hyperspectral image classification technology, classification methods based on machine learning, such as support vector machine (SVM), decision tree (DT), and random forest (RF), usually process the spectral features of hyperspectral images, but ignore the spatial features of the images, resulting in unsatisfactory classification results. In addition, most of these methods require manual feature extraction, over-reliance on manual experience and time-consuming, making it difficult for traditional methods to play their due role in many situations.

[0004] With the popularity of convolutional neural networks (CNN), hyperspectral image classification methods based on CNN continue to emerge. In addition to CNN, some network categories with excellent performance have been used for hyperspectral image classification. Transformer was originally a model for natural language processing, which can capture global dependencies in input sequences through the self-attention mechanism. In recent years, Transformer-based network models have also been widely used in the field of image processing. However, although CNN alone is good at extracting local features, due to the limitations of the convolution kernel, the receptive field is often limited to the area of ​​the convolution kernel size, so it performs poorly in global modeling. Transformers perform well in solving the long-term dependency problem of spectra and modeling global information, but they lose the ability to capture local context or semantic features. Summary of the invention

[0005] The main problems solved by the present invention are the complexity of the hyperspectral image classification model and the insufficient feature extraction; and the low efficiency of CNN-based and Transformer-based methods.

[0006] The technical solution adopted by the present invention is: in the first aspect, the present invention provides a hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer, comprising the following steps:

[0007] Step 1: Acquire hyperspectral image data and perform preprocessing;

[0008] Step 2: Input the preprocessed data in step 1 into a dual-branch multi-scale CNN module; the dual-branch multi-scale CNN module includes 3DCNN and 2DCNN branches; the 3DCNN branch simultaneously extracts the spatial features and spectral features of the hyperspectral image and captures features of different scales, and the 2DCNN branch extracts the local spatial features of the hyperspectral image and the detailed distinction between adjacent spectra; the results of the two branches are fused and merged as output;

[0009] Step 3: Input the output data after step 2 into the memory modules MT1 and MT2; MT1 performs channel-by-channel and point-by-point convolution on the output data after step 2 to obtain P MT1 MT2 performs average pooling and maximum pooling on the output data after step 2 and then concatenates them to obtain P MT2 ;

[0010] Step 4: Combine the output data of step 2 and the P after step 3 MT1 and P MT2 The data are used as the input of the memory-enhanced Transformer for training; the memory-enhanced Transformer includes two sublayers in series, the first sublayer includes the Layernorm layer and the memory-enhanced layer MSA; the output data of step 2 is passed through the Layernorm layer of the first sublayer to obtain P n , the memory enhancement layer MSA is used to convert P MT1 and P MT2 After linear transformation and concatenation with K and V of the attention mechanism, the attention mechanism is calculated, and then P n After addition, the results are taken as the output of the first sub-layer and passed to the second sub-layer for calculation. The second sub-layer includes the Layernorm layer and the feed-forward layer. The output of the feed-forward layer plus the output of the first sub-layer are taken as the final output of the second sub-layer.

[0011] Step 5: Input the output result of the second sub-layer in step 4 into the mlp-head module to obtain the final classification result.

[0012] Furthermore, the preprocessing method in step 1 is specifically as follows:

[0013] Step 1.1: Input the raw hyperspectral data with the size of X∈R H×W×B, where H is the height of the hyperspectral image, W is the width of the hyperspectral image, and B is the dimension of the hyperspectral image; the hyperspectral raw data is subjected to principal component analysis and dimensionality reduction to obtain the hyperspectral data X1∈R H×W×C , C represents the dimension of the hyperspectral image after dimensionality reduction;

[0014] Step 1.2, extract cube data patches with the center pixel as label from X1, the size of each patch is represented as P∈R S×S×C , where S×S represents the width and height of the patch, which are the same size.

[0015] Furthermore, the dual-branch multi-scale CNN module in step 2 is as follows:

[0016] The dual-branch multi-scale CNN module includes a 3DCNN branch, a 2DCNN branch, and a 2D convolution; the 3DCNN contains two 3D convolution layers with different convolution kernel sizes, which can simultaneously extract the spatial and spectral features of hyperspectral images and capture features of different scales, specifically expressed as:

[0017] P 3D =Conv3D2(Conv3D1(P))

[0018] Where P represents the input data, Conv3D1 represents the 3D convolution layer with a larger convolution kernel, and Conv3D2 represents the 3D convolution layer with a smaller convolution kernel;

[0019] 2DCNN contains three 2D convolution layers with different convolution kernel sizes, which can be used to extract the local spatial features of hyperspectral images and the detailed differences between adjacent spectra, specifically expressed as:

[0020] P 2D =Conv2D3(Conv2D2(Conv2D1(P)))

[0021] Where P represents the input data, Conv2D1, Conv2D2 and Conv2D3 represent three 2D convolution operations, and the convolution kernel sizes of the three convolution layers increase in sequence;

[0022] The data obtained by the 3DCNN branch and the 2DCNN branch are merged and the result is input into the last 2D convolutional layer, which is specifically expressed as:

[0023]

[0024] in represents element addition, r1 and r2 represent weight ratios, which are used to adjust the weight ratio of the 3D branch and the 2D branch.

[0025] Furthermore, step three is as follows:

[0026] The outputs obtained in step 2 are input into memory modules MT1 and MT2 respectively. The MT1 module contains a channel-by-channel convolution layer, a point-by-point convolution layer, and a 2D convolution layer; the MT2 module contains a maximum pooling layer and an average pooling layer, which are specifically expressed as:

[0027] P MT1 =Conv2D(Conv2D p (Conv2D c (P s2 )))

[0028] Where P s2 Represented as the data after step 2, Conv2D c Represented as a channel-by-channel convolutional layer in MT1, Conv2D p The layer is represented as a point-wise convolutional layer in MT1;

[0029] P MT2 =Concat(AvgPool(P s2 )), MaxPool(P s2 )))

[0030] Where P s2 It represents the data after step 2, AvgPool represents the average pooling layer in MT2, and MaxPool layer represents the maximum pooling layer in MT2.

[0031] Furthermore, the memory-enhanced Transformer in step 4 is as follows:

[0032] Step 4.1, the memory-enhanced Transformer consists of two sub-layers in series. The first sub-layer contains a Layernorm layer, a memory-enhanced layer MSA and a residual connection; the second layer contains a Layernorm layer, a feed-forward layer and a residual connection; first, the output obtained in step 2 is embedded with position information and concatenated with an all-zero vector as a learnable classification vector; then the result is passed to the Layernorm layer to obtain the data P n , and take the result of step 3 as P MT1 and P MT2 ;

[0033] Step 4.2, at this time [P n ,P MT1 ,P MT2 ] is used as the input of the memory enhancement layer MSA. First, the feature vector P n and memory vector P MT1 ,PMT2 Perform linear transformation to obtain 5 matrices (W q ,W k ,W v ,W m1 ,W m2 ),W q The matrix is ​​represented as the query matrix, W k Expressed as a bond matrix, W v Represented as a value matrix, W m1 ,W m2 Represented as a memory matrix;

[0034] Q=P n W q

[0035] K=P n W K

[0036] V=P n W V

[0037] M1=P MT1 W m1

[0038] M2=P MT2 W m2

[0039] Then the MSA module splices K with M1 and V with M2 to reduce the loss of shallow information, strengthen the communication between information, and enhance memory;

[0040] K′=concat(K,M1)

[0041] V′=concat(V,M2)

[0042] Finally, the attention mechanism is calculated:

[0043]

[0044] Where Q is the query, K' is the key and V' is the value, T represents the transpose operation, d k Indicates the dimension size of K';

[0045] Step 4.3, P n Add the data obtained in step 4.2, first pass it to the Layernorm layer, and then pass the data to the feed-forward layer;

[0046] F forward =Linear(Gelu(Linear(x)))

[0047] So the overall calculation process of the memory-enhanced Transformer is:

[0048] P t1 =MSA(LN(P n ))+P n

[0049] P t2 =feed-forward(LN(P t1 ))+P t1

[0050] Where P t1 Indicates P n Data after the first layer of memory-enhanced Transformer, P t2 Indicates P t1 Data after the second layer of the memory-augmented Transformer.

[0051] Furthermore, the mlp-head module in step 5 consists of a LayerNorm layer and a Linear layer:

[0052] P m =Linear(LN(P t2 ))

[0053] Where LN represents the LayerNorm layer, Linear represents the Linear layer, and P m is the final classification result.

[0054] In a second aspect, the present invention also provides a hyperspectral image classification device based on a dual-branch multi-scale CNN and a memory-enhanced Transformer, comprising a memory and one or more processors, wherein the memory stores an executable code, and when the processor executes the executable code, the hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer is implemented.

[0055] In a third aspect, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer.

[0056] In a fourth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer.

[0057] The beneficial effects of the present invention are as follows:

[0058] 1. The present invention improves and combines the CNN structure and the Transformer structure, combining the advantages of the two structures to improve the accuracy and efficiency of hyperspectral image classification;

[0059] 2. The dual-branch CNN feature extraction module in the present invention uses a 3DCNN branch and a 2DCNN branch, respectively, and fully combines and extracts the spatial features and spectral-spatial joint features of the hyperspectral spectrum;

[0060] 3. The memory modules MT1 and MT2 in the present invention are simple and efficient, and can extract valid data to the maximum extent in a simple structure. MT1 performs detailed and global memory on the data, and MT2 performs important feature memory and fuzzy memory on the data.

[0061] 4. The memory-enhanced Transformer in the present invention fully strengthens the exchange of information and reduces the loss of features. It is the main structure of the model and greatly improves the accuracy and performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of the method of the present invention;

[0063] Figure 2 This is a schematic diagram of the overall network structure of the present invention;

[0064] Figure 3 This is a schematic diagram of the dual-branch multi-scale CNN network structure of the present invention;

[0065] Figure 4 This is a schematic diagram of the memory-enhanced Transformer network structure of the present invention;

[0066] Figure 5 is the classification result of the method used in the Pavia University dataset in the method of the present invention;

[0067] Figure 6 is the classification result of the method used in the WHU-Hi-HanChuan dataset in the method of the present invention;

[0068] Figure 7 This is a structural diagram of a hyperspectral image classification device based on a dual-branch multi-scale CNN and a memory-enhanced Transformer provided by the present invention. DETAILED DESCRIPTION

[0069] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] In order to solve the problems of complex hyperspectral image classification models, insufficient feature extraction, and low efficiency of CNN and Transformer, this paper proposes a hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer. Figure 1 This is the flow chart of the present invention, and the overall network structure is as follows Figure 2 The specific workflow is as follows:

[0071] (1) Acquire hyperspectral image data and perform preprocessing;

[0072] (1-1) The hyperspectral image with the original data size is X∈R H×W×B , where H is the height of the hyperspectral image, W is the width of the hyperspectral image, and B is the dimension of the hyperspectral image. Principal component analysis of the raw hyperspectral data can be used to reduce the dimension by retaining the main dimensional features and removing some useless noise to obtain the hyperspectral data X1∈R H×W×C , C represents the dimension of the hyperspectral image after dimensionality reduction.

[0073] (1-2) Extract cube data patches with the central pixel as the label from X1. The size of each patch is represented by P∈R S×S×C , where S×S represents the width and height of the patch, which are the same size. And the data set and test set are divided.

[0074] (2) Process the data preprocessed in step (1) and input it into the hyperspectral image feature extraction module (MCNN) of the dual-branch multi-scale CNN to extract spectral-spatial features and local spatial features of the hyperspectral data. The method includes the following steps:

[0075] (2-1) The dual-branch multi-scale CNN network structure of the present invention is as follows Figure 3 As shown. The data after step (1) is input into the 3DCNN branch in MCNN. 3DCNN contains two 3D convolution layers with different convolution kernel sizes, which can simultaneously extract the spatial features and spectral features of hyperspectral images and capture features of different scales. The specific representation is:

[0076] P 3D =Conv3D2(Conv3D1(P))

[0077] Among them, Conv3D1 represents a 3D convolution layer with a larger convolution kernel, and Conv3D2 represents a 3D convolution layer with a smaller convolution kernel.

[0078] (2-2) Input the data into the 2DCNN branch of MCNN. 2DCNN contains three 2D convolution layers with different convolution kernel sizes. The convolution kernel sizes of the three convolution layers increase in sequence, which can be used to extract the local spatial features of hyperspectral images and the detailed differences between adjacent spectra. Specifically expressed as:

[0079] P 2D =Conv2D3(Conv2D2(Conv2D1(P)))

[0080] (2-3) Input the data after step (1) into the 3DCNN branch and the 2DCNN branch respectively to merge the data obtained, and input the result into the last 2D convolution layer. Specifically expressed as:

[0081]

[0082] in represents element addition. r1 and r2 represent weight ratios, which are used to adjust the weight ratio of the 3D branch and the 2D branch.

[0083] (3) The outputs obtained in step (2) are input into memory modules MT1 and MT2 respectively. The MT1 module contains a channel-by-channel convolution layer, a point-by-point convolution layer and a 2D convolution layer. The MT2 module contains a maximum pooling layer and an average pooling layer. The effect of the MT1 memory module is similar to the detailed memory features in real life. The maximum pooling layer in MT2 is similar to the important memory features in real life, and the average pooling layer is similar to the fuzzy memory features in real life. The structures of MT1 and MT2 are specifically expressed as:

[0084] P MT1 =Conv2D(Conv2D p (Conv2D c (P s2 )))

[0085] Where P s2 Represented as the data after step (2), Conv2D c Represented as a channel-by-channel convolutional layer in MT1, Conv2D p The layers are represented as point-wise convolutional layers in MT1.

[0086] P MT2 =Concat(AvgPool(P s2 ), MaxPool(P s2 ))

[0087] Where P s2It represents the data after step (2), AvgPool represents the average pooling layer in MT2, and MaxPool layer represents the maximum pooling layer in MT2.

[0088] (4) The data output in step (3) is concatenated with K and V in the memory-enhanced Transformer. The network structure of the memory-enhanced Transformer is as follows: Figure 4 The specific method of step (4) comprises the following steps:

[0089] (4-1) The memory-enhanced Transformer consists of two sub-layers connected in series. The first sub-layer contains a Layernorm layer, a memory-enhanced layer MSA, and a residual connection. The second layer contains a Layernorm layer, a feed-forward layer, and a residual connection. First, the output obtained in step (2) is embedded with position information and concatenated with a full zero vector as a learnable classification vector. The result is then passed to the Layernorm layer to obtain the data P n , the result obtained in step (3) is taken as P MT1 and P MT2 .

[0090] (4-2) At this time [P n ,P MT1 ,P MT2 ] is used as the input of the memory enhancement layer MSA. First, the feature vector P n and memory vector P MT1 ,P MT2 Perform linear transformation to obtain 5 matrices (W q ,W k ,W v ,W m1 ,W m2 ),W q The matrix is ​​represented as the query matrix, W k Expressed as a bond matrix, W v Represented as a value matrix, W m1 ,W m2 Represented as a memory matrix.

[0091] Q=P n W q

[0092] K=P n W K

[0093] V=P n W V

[0094] M1=P MT1 Wm1

[0095] M2=P MT2 W m2

[0096] Then, in the memory enhancement layer MSA, K is concatenated with M1, and V is concatenated with M2. This can reduce the loss of shallow information, strengthen the information exchange between the shallow structure and the deep structure, and enhance information memory.

[0097] K′=concat(K,M1)

[0098] V′=concat(V,M2)

[0099] Finally, the attention mechanism is calculated:

[0100]

[0101] Where Q is the query, K' is the key and V' is the value, T represents the transpose operation, d k Indicates the dimension size of K'.

[0102] (4-3) P n Add the data obtained by (4-2) and pass it to the Layernorm layer first, and then pass the data to the feed-forward layer. The calculation process of the feed-forward layer is:

[0103] F forward =Linear(Gelu(Linear(x)))

[0104] So the overall calculation process of the memory-enhanced Transformer is:

[0105] P t1 =MSA(LN(P n ))+P n

[0106] P t2 =feed-forward(LN(P t1 ))+P t1

[0107] Where P t1 Indicates P n Data after the first layer of memory-enhanced Transformer. t2 Indicates P t1 Data after the second layer of the memory-augmented Transformer.

[0108] (5) The data processed by the memory-enhanced Transformer is transferred to the mlp-head layer for final result classification.

[0109] P m =Linear(LN(P t2 ))

[0110] Where LN represents the Layernorm layer, Linear represents the Linear layer, and P m Indicates the final classification result.

[0111] The effect of the method of the present invention is further confirmed by the following experiments:

[0112] The method evaluation included overall accuracy (OA), average accuracy (AA), and Kappa coefficient (KAPPA).

[0113] Overall accuracy (OA) indicates the proportion of correctly classified samples to the total number of classified samples;

[0114] Average accuracy (AA) represents the average classification accuracy of each category;

[0115] The Kappa coefficient (KAPPA) is used to measure the consistency of a classification model, taking into account the difference between the proportion of correct predictions made by the model and the accuracy of random guessing alone;

[0116] The patch_size is 13*13, the training sample of Pavia University is 0.5%, and the training sample of WHU-Hi-HanChuan is 1%.

[0117] The classification results on the Pavia University and WHU-Hi-HanChuan datasets are as follows: Figure 5 , Figure 6 shown.

[0118] Table 1 Comparison of the accuracy of each classification method at Pavia University

[0119]

[0120]

[0121] Table 2 Comparison of the accuracy of each classification method of WHU-Hi-HanChuan

[0122]

[0123]

[0124] As can be seen from Table 1 and Table 2, the classification accuracy of the present invention in various categories of Pavia University and WHU-Hi-HanChuan datasets is higher than that of other existing methods. And the accuracy of OA, AA, and KAPPA are also higher than that of other existing classification methods. In Pavia University, the OA of the present invention is 5.51%, 4.09%, 2.43%, and 2.98% higher than that of HyBridSN, RSSAN, SSFTT, and morphFormer respectively; AA is 9.02%, 6.95%, 3.14%, and 4.57% higher respectively. KAPPA is 7.37%, 5.49%, 3.25%, and 3.95% higher respectively; and as can be seen from Table 1, the accuracy of the network structure model based on Transformer on Pavia University is better than that of the network structure model based on CNN. In the WHU-Hi-HanChuan dataset, the OA of the present invention is 1.4%, 4.18%, 1.49%, and 1.16% higher than that of HyBridSN, RSSAN, SSFTT, and morphFormer respectively; AA is 4.87%, 8.43%, 2.82%, and 2.16% higher respectively. KAPPA is 1.64%, 4.89%, 1.74%, and 1.36% higher respectively; Table 2 shows that in the WHU-Hi-HanChuan dataset, the accuracy of the first few categories of HyBridSN is higher than that of the Transformer-based network structure model. However, the overall accuracy of the CNN-based network structure model is still lower than that of the Transformer-based network structure model.

[0125] Corresponding to the aforementioned embodiment of a hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer, the present invention also provides an embodiment of a hyperspectral image classification device based on a dual-branch multi-scale CNN and a memory-enhanced Transformer.

[0126] See also Figure 7 A hyperspectral image classification device based on a dual-branch multi-scale CNN and a memory-enhanced Transformer provided in an embodiment of the present invention comprises a memory and one or more processors, wherein the memory stores executable codes, and when the processor executes the executable codes, it is used to implement a hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer in the above-mentioned embodiment.

[0127] The embodiment of a hyperspectral image classification device based on a dual-branch multi-scale CNN and a memory-enhanced Transformer provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From a hardware perspective, if Figure 7 As shown in FIG. 1 , a hardware structure diagram of a device having data processing capability for a hyperspectral image classification device based on dual-branch multi-scale CNN and memory-enhanced Transformer provided by the present invention is shown in FIG. Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0128] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0129] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0130] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer in the above embodiment is implemented.

[0131] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0132] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer.

[0133] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer, characterized in that: The following steps are involved: Step 1: Acquire hyperspectral image data and perform preprocessing; Step 2: Input the preprocessed data in step 1 into a dual-branch multi-scale CNN module; the dual-branch multi-scale CNN module includes 3DCNN and 2DCNN branches; the 3DCNN branch simultaneously extracts the spatial features and spectral features of the hyperspectral image and captures features of different scales, and the 2DCNN branch extracts the local spatial features of the hyperspectral image and the detailed distinction between adjacent spectra; the results of the two branches are fused and merged as output; Step 3, input the output data after step 2 into the memory modules MT1 and MT2; MT1 performs channel-by-channel and point-by-point convolution on the output data after step 2 to obtain P MT1 MT2 performs average pooling and maximum pooling on the output data after step 2 and then concatenates them to obtain P MT2 ; Step 4: Combine the output data of step 2 and the P after step 3 MT1 and P MT2 The data are used as the input of the memory-enhanced Transformer for training; the memory-enhanced Transformer includes two sublayers in series, the first sublayer includes the Layernorm layer and the memory-enhanced layer MSA; the output data of step 2 is passed through the Layernorm layer of the first sublayer to obtain P n , the memory enhancement layer MSA is used to convert P MT1 and P MT2 After linear transformation and concatenation with K and V of the attention mechanism, the attention mechanism is calculated, and then P n After addition, the results are taken as the output of the first sub-layer and passed to the second sub-layer for calculation. The second sub-layer includes the Layernorm layer and the feed-forward layer. The output of the feed-forward layer plus the output of the first sub-layer are taken as the final output of the second sub-layer. Step 5: Input the output result of the second sub-layer in step 4 into the mlp-head module to obtain the final classification result.

2. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 1 is characterized in that: The preprocessing method in step 1 is as follows: Step 1.1: Input the raw hyperspectral data with the size of X∈R H×W×B , where H is the height of the hyperspectral image, W is the width of the hyperspectral image, and B is the dimension of the hyperspectral image; the hyperspectral raw data is subjected to principal component analysis and dimensionality reduction to obtain the hyperspectral data X1∈R H×W×C , C represents the dimension of the hyperspectral image after dimensionality reduction; Step 1.2, extract cube data patches with the center pixel as label from X1, and the size of each patch is represented as P∈R S×S×C , where S×S represents the width and height of the patch, which are the same size.

3. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 1 is characterized in that: The dual-branch multi-scale CNN module in step 2 is as follows: The dual-branch multi-scale CNN module includes a 3DCNN branch, a 2DCNN branch, and a 2D convolution; the 3DCNN contains two 3D convolution layers with different convolution kernel sizes, which can simultaneously extract the spatial and spectral features of hyperspectral images and capture features of different scales, specifically expressed as: P 3D =Conv3D2(Conv3D1(P)) Where P represents the input data, Conv3D1 represents the 3D convolution layer with a larger convolution kernel, and Conv3D2 represents the 3D convolution layer with a smaller convolution kernel; 2DCNN contains three 2D convolution layers with different convolution kernel sizes, which can be used to extract the local spatial features of hyperspectral images and the detailed differences between adjacent spectra, specifically expressed as: P 2D =Conv2D3(Conv2D2(Conv2D1(P))) Where P represents the input data, Conv2D1, Conv2D2 and Conv2D3 represent three 2D convolution operations, and the convolution kernel sizes of the three convolution layers increase in sequence; The data obtained by the 3DCNN branch and the 2DCNN branch are merged and the result is input into the last 2D convolutional layer, which is specifically expressed as: in represents element addition, r1 and r2 represent weight ratios, which are used to adjust the weight ratio of the 3D branch and the 2D branch.

4. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 1 is characterized in that: Step three is as follows: The outputs obtained in step 2 are input into memory modules MT1 and MT2 respectively. The MT1 module contains a channel-by-channel convolution layer, a point-by-point convolution layer, and a 2D convolution layer; the MT2 module contains a maximum pooling layer and an average pooling layer, which are specifically expressed as: P MT1 =Conv2D(Conv2D p (Conv2D c (P s2 ))) Where P s2 Represented as the data after step 2, Conv2D c Represented as a channel-by-channel convolutional layer in MT1, Conv2D p The layer is represented as a point-wise convolutional layer in MT1; P MT2 =Concat(AvgPool(P s2 )),MaxPool(P s2 ))) Where P s2 It represents the data after step 2, AvgPool represents the average pooling layer in MT2, and MaxPool layer represents the maximum pooling layer in MT2.

5. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 1 is characterized in that: The memory-enhanced Transformer in step 4 is as follows: Step 4.1, the memory-enhanced Transformer consists of two sub-layers in series. The first sub-layer contains a Layernorm layer, a memory-enhanced layer MSA and a residual connection; the second layer contains a Layernorm layer, a feed-forward layer and a residual connection; first, the output obtained in step 2 is embedded with position information and concatenated with an all-zero vector as a learnable classification vector; then the result is passed to the Layernorm layer to obtain the data P n , and take the result of step 3 as P MT1 and P MT2 ; Step 4.2, at this time [P n ,P MT1 ,P MT2 ] is used as the input of the memory enhancement layer MSA. First, the feature vector P n and memory vector P MT1 ,P MT2 Perform linear transformation to obtain 5 matrices (W q ,W k ,W v ,W m1 ,W m2 ),W q The matrix is ​​represented as the query matrix, W k Expressed as a bond matrix, W v Represented as a value matrix, W m1 ,W m2 Represented as a memory matrix; Q=P n W q K=P n W K V=P n W V M1=P MT1 W m1 M2=P MT2 W m2 Then the MSA module splices K with M1 and V with M2 to reduce the loss of shallow information, strengthen the communication between information, and enhance memory; K′=concat(K,M1) V′=concat(V,M2) Finally, the attention mechanism is calculated: Where Q is the query, K' is the key and V' is the value, T represents the transpose operation, d k Indicates the dimension size of K'; Step 4.3, P n Add the data obtained in step 4.2, first pass it to the Layernorm layer, and then pass the data to the feed-forward layer; F forward =Linear(Gelu(Linear(x))) So the overall calculation process of the memory-enhanced Transformer is: P t1 =MSA(LN(P n ))+P n P t2 =feed-forward(LN(P t1 ))+P t1 Where P t1 Indicates P n Data after the first layer of memory-enhanced Transformer, P t2 Indicates P t1 Data after the second layer of the memory-augmented Transformer.

6. The hyperspectral image classification method based on dual-branch multi-scale CNN and memory-enhanced Transformer according to claim 5 is characterized in that: The mlp-head module in step 5 consists of a LayerNorm layer and a Linear layer: P m =Linear(LN(P t2 )) Where LN represents the LayerNorm layer, Linear represents the Linear layer, and P m is the final classification result.

7. A hyperspectral image classification device based on dual-branch multi-scale CNN and memory-enhanced Transformer, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer is implemented as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer as described in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a hyperspectral image classification method based on a dual-branch multi-scale CNN and a memory-enhanced Transformer is implemented as described in any one of claims 1 to 6.

Images