A hyperspectral image classification method based on reparameterized MLP

Abstract

The invention discloses a hyperspectral image classification method based on a reparameterized MLP, which belongs to the technical field of image classification. The method sequentially performs the following steps: inputting a hyperspectral image, obtaining a sample set, determining a training sample set and a test sample set, slicing the hyperspectral samples, constructing a reparameterized MLP model, and training the reparameterized MLP model with training data to classify the hyperspectral image. The invention replaces part of the convolutional layers with a fully connected layer to provide a global representation capability and a position perception capability, and finally reparameterizes the convolutional layer and the fully connected layer into a fully connected layer through a reparameterization method, thereby giving the fully connected layer the ability to capture local information that it does not have, and improving the classification accuracy and reasoning speed when classifying the hyperspectral image, and providing a new idea for subsequent deep learning algorithms in hyperspectral image classification research and engineering applications.

Description

Technical Field

[0001] The invention discloses a hyperspectral image classification method based on re-parameterized MLP, belonging to the technical field of image classification. Background Art

[0002] Hyperspectral remote sensors are able to capture hyperspectral images with rich spectral and spatial information, and can accurately characterize and identify different land covers. Since hyperspectral images are rich in multiple sources of information, they have broad application prospects in many fields, including crop classification and water quality condition detection in agriculture, product quality in the food industry, water quality analysis and sea ice detection in marine resource management, forest health in forestry and environmental management, and pest analysis in plantations, artificial material identification in security and defense applications, and 3D reconstruction and image detection in visual technology. Hyperspectral image classification refers to the task of assigning a category to each pixel in the scene. Since the fields involved in hyperspectral images are wide and far-reaching, hyperspectral image classification is a basic and hot issue in the remote sensing community.

[0003] In recent years, various studies applied to hyperspectral image classification have demonstrated the success of deep learning-based methods. Among the hyperspectral image classification methods based on deep learning, the Convolutional Neural Network (CNN) with local connections and shared weight structures has become the mainstream method for hyperspectral image classification. However, since pixels of the same category in a hyperspectral image may have different spectral features (same object, different spectra), and pixels of different categories may have the same spectral features (same spectrum, different objects), it is difficult to distinguish the categories represented by different pixels using only spectral features. This causes the convolutional neural network to be unable to fully utilize the rich semantic attributes around specific pixels in hyperspectral images, resulting in the loss of a large amount of hyperspectral feature information. In addition, when the convolutional neural network is used to classify hyperspectral images, due to the poor training efficiency and generalization ability of the convolutional neural network classification model, it leads to problems such as weak processing ability for hyperspectral data and low classification accuracy. Summary of the invention

[0004] To solve the above problems, the present invention proposes a hyperspectral image classification method based on reparameterized MLP. By proposing a reparameterized MLP model composed of multiple fully connected layers for hyperspectral image classification, the global representation capability of the fully connected layers and the local capture characteristics of the convolutional layers are fully utilized, thereby achieving a better classification accuracy than traditional CNN in hyperspectral image classification.

[0005] The object of the present invention is achieved in that:

[0006] A hyperspectral image classification method based on reparameterized MLP includes the following steps:

[0007] Step a: Input hyperspectral image

[0008] Input a hyperspectral image to be classified, and set each pixel point in the input hyperspectral image as a sample;

[0009] Step b: Obtain sample set

[0010] For the coordinate positions of all samples of the hyperspectral image, a coordinate transformation method is used to arrange the samples of each dimension of the hyperspectral image into a row vector to form a two-dimensional matrix, thereby obtaining a sample set of the spectral domain of the hyperspectral image; each column in the two-dimensional matrix is ​​a sample;

[0011] Where S is the number of pixels in a hyperspectral image with m rows and n columns, S = m × n, y sp represents the Sth pixel of the pth band; p represents the number of bands of the hyperspectral image;

[0012]

[0013] Step c: Determine the training sample set and the test sample set

[0014] Using the method of equal probability sampling, 10% of the samples in the sample set of the spectral domain of the hyperspectral image are randomly selected as the training sample set, and the coordinate positions of the training sample set of the spectral domain of the hyperspectral image are recorded; the remaining 90% of the samples are used as the test sample set of the spectral domain of the hyperspectral image;

[0015] Step d: Slicing of hyperspectral samples

[0016] Each pixel in the hyperspectral spectrum is processed as a fixed window to generate a single sample;

[0017] Step e: Build a reparameterized MLP model and train the reparameterized MLP model using training data

[0018] The re-parameterized MLP model includes a training phase and a testing phase; the training phase consists of three parts: a global perceptron module, a partitioned perceptron module, and a local perceptron module; and the re-parameterization is subsequently performed in the testing phase;

[0019] Step f: Classify the hyperspectral image

[0020] The hyperspectral feature map is fed into the Softmax function in the discriminator to finally complete the classification of the hyperspectral image and finally output the hyperspectral classification result map.

[0021] In the above-mentioned hyperspectral image classification method based on reparameterized MLP, step e comprises the following steps:

[0022] Step e1: The global perceptron module includes two branches. Branch 1 first divides the input hyperspectral feature map, and branch 2 averages the original feature map, then sends the average pooled hyperspectral feature map to the BN layer plus two fully connected layers, and then adds it to the hyperspectral feature map obtained after the branch is divided; after completing the global perceptron operation, the obtained features are sent to the subsequent partitioned perceptron and local perceptron as the input of the partitioned perceptron and the local perceptron;

[0023] Step e2: The partitioned perceptron module contains a BN layer and a group convolution;

[0024] Step e3: The local perceptron module contains convolution layers and BN layers with kernel sizes of 1, 3, 5, and 7. In order to reduce the number of parameters, grouped convolution is used, and the number of groups is consistent with that of the partitioned perceptron. First, the local perceptron module simultaneously sends the segmented hyperspectral feature map to the convolution layers with kernel sizes of 1, 3, 5, and 7. For the subsequent hyperspectral feature addition operation, the convolution layers need to be edge-filled. Padding is to increase the number of pixels on each edge. Then the feature map is sent to the BN layer, and the outputs of all convolution branches are added to the output of the partitioned perceptron as the final output.

[0025] Step e4: During the test phase, reparameterization is then performed, which is a process of fusing the convolutional weights into the fully connected layer; during the test phase, the local perceptron module and the partitioned perceptron module are fused into a fully connected layer.

[0026] Furthermore, step e1 is specifically as follows:

[0027] The global perceptron module includes two branches. Branch 1 first divides the input hyperspectral feature map to achieve parameter sharing of different partition features. After division, the hyperspectral feature map changes from 9×9×200 to 3×3×200. However, simple division will destroy the correlation between different partitions of the same channel. In order to make each partition relevant, the global perceptron adds branch 2 to process the hyperspectral features.

[0028] Branch 2 pairs of original features Fig. 9 ×9×200 for average pooling, the size of the hyperspectral feature map becomes 3×3×200, and the tensor R C×H×W represents the hyperspectral feature map, where C represents the number of channels, H and W represent the height and width of the input hyperspectral feature map, respectively; h and w represent the height and width of the hyperspectral feature map after segmentation, respectively; after calculating h 1 =3,w 1 =3, the calculation formula is h 1 =H / h and w 1 =W / w;

[0029] Then the hyperspectral features Figure 3 ×3×200 is fed into the BN layer and two fully connected layers. For the fully connected layers, P and Q represent the input and output dimensions respectively, V (in) and V (out) Represent the input and output of the fully connected layer network respectively; the fully connected kernel W∈R Q×P ;

[0030] Matrix multiplication (MMUL) is the output V of the fully connected layer (out) The calculation formula is:

[0031] V (out) =MMUL(V (in) ,W)=V (in) ·W T

[0032] After the BN layer and two fully connected layers, the hyperspectral vector is transformed into 1×1×200; then it is added to the hyperspectral feature map obtained after branch segmentation; after completing the global perceptron operation, the obtained features are sent to the subsequent partition perceptron and local perceptron as the input of the partition perceptron and local perceptron.

[0033] Furthermore, step e2 is specifically as follows:

[0034] The partitioned perceptron module contains a BN layer and a group convolution; the input of the partitioned perceptron is 3×3×200, and then connected to a 1×1, groups=4 group convolution and BN layer; after the BN layer and a group convolution, the input 3×3×200 becomes the original hyperspectral feature input 9×9×200;

[0035] in represents the output feature; p is the number of filled pixels; F∈R C / g×K×K is the convolution kernel; g represents the number of convolution groups; M (out) The calculation formula is: (out) =gCONV(M (in) ,F,g,p)

[0036] Among them, the group convolution writes the group matrix multiplication (gMMUL) as:

[0037] M'=RS(V (in) ,(N,p,1,1)),F'=RS(W,p / g,l,l)

[0038] gCONV(V (in) ,W,g)=RS(gCONV(M',F',g,O),(N,Q)).

[0039] Furthermore, step e3 is specifically as follows:

[0040] The local perceptron module contains convolutional layers and BN layers with kernel sizes of 1, 3, 5, and 7. In order to reduce the number of parameters, group convolution is used. The number of groups is consistent with that of the partitioned perceptron. First, the local perceptron module takes the segmented hyperspectral features Figure 3 ×3×200 are simultaneously sent to the convolutional layers with kernel sizes of 1, 3, 5, and 7; for the subsequent hyperspectral feature addition operation, the convolutional layers need to be edge-filled; padding is to increase the number of pixels on each edge in order to keep the feature map not too small; then the feature map is sent to the BN layer, and the output of all convolutional branches is added to the output of the partitioned perceptron 3×3×200 as the final output;

[0041] During the training phase, RepMLP adds the output of the convolution to the output of the FC. This has the following benefits:

[0042] First, reduce FLOPs and increase speed. After converting all convolutions to equivalent FC kernels, due to the additivity of matrix multiplication AX+BX=(A+B)X, a sparse and shared parameter FC plus a non-sparse and non-shared parameter FC can be equivalently converted to one FC. The parameter of the equivalent conversion FC is the sum of the two added matrices. In this way, these convolutions can be equivalently removed. This idea also belongs to structural reparameterization.

[0043] Second, with the same number of parameters, the FLOPs of FC is much lower than that of traditional convolution.

[0044] Furthermore, step e4 is specifically as follows:

[0045] During the test phase, reparameterization is then performed, which is a process of fusing the convolutional weights into the fully connected layer; during the test phase, the local perceptron module and the partitioned perceptron module are fused into a fully connected layer;

[0046] Since matrix multiplication is additive, the addition of the fully connected layer of the partitioned perceptron module and the convolutional layer of the local perceptron module is written as: MMUL(M (in) ,W (1) )+MMUL(M (in) ,W (2) )=MMUL(M (in) ,W (1) +W (2) )

[0047] Convolution is a sparse matrix multiplication, so there is an equivalent weight transformation, such as:

[0048] MMUL(M (in) ,W (1) )+MMUL(M (in),W (F,p) )=CONV(M (in) ,F,p)

[0049] Because in matrix multiplication, if you insert an identity matrix into it, it will not change the matrix result, so we have:

[0050] V out =V in *(I,W (F,p) )

[0051] Where I represents the identity matrix;

[0052] I=W (F,p)

[0053] It can be regarded as the result of convolution kernel F on the unit matrix;

[0054] For any input M (in) , convolution kernel F, there exists an FC kernel that satisfies:

[0055] M (out) =CONV(M (in) ,F,p)=MMUL(M (in) ,W (F,p) )

[0056] Using matrix multiplication, we have:

[0057] V out =V in ·W (F,p)τ

[0058] Insert the identity matrix I(Chw,Chw)) into it and apply the following law:

[0059] V out =V in ·(I·W (F,p)τ )

[0060] Among them, W (F,p) is constructed from F, I·W (F,p)τ Represents the feature M (I) Perform convolution, then we have:

[0061] M (I) =CONV(I,(Chw,C,h,w))

[0062] I.W. (F,p)τ =CONV(M (in) ,F,p)

[0063] V out =V in ·(I*W (F,p)τ,(Chw,Ohw))

[0064] By comparing the above formula, we can get:

[0065] W (F,p) =CONV(M (I) ,F,p),(Chw,Ohw)

[0066] The above formula shows how to construct W from F and p (F,p) ; In short, the equivalent FC kernel of the convolution kernel can be obtained by convolving the identity matrix.

[0067] Beneficial effects:

[0068] Traditional CNN expands the receptive field by stacking convolutional layers to obtain global information. The classification model used in this method has poor flexibility. When processing high-dimensional hyperspectral remote sensing images with a small number of training samples, the classification accuracy may decrease with the increase of data dimension. The present invention proposes a hyperspectral image classification method with a reparameterized MLP, and attempts to replace part of the convolutional layers with fully connected layers to provide global representation capabilities and position perception capabilities. Finally, the convolutional layers and fully connected layers are reparameterized into a fully connected layer through a reparameterization method, which gives the fully connected layer the ability to capture local information that it does not have. This improves the classification accuracy and reasoning speed when classifying hyperspectral images, and provides new ideas for subsequent deep learning algorithms in hyperspectral image classification research and engineering applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of the hyperspectral image classification method of the present invention with re-parameterized MLP.

[0070] Figure 2 It is the overall block diagram of the hyperspectral image classification method of the present invention with re-parameterized MLP.

[0071] Figure 3 It is a diagram of the three-layer MLP structure in the method of the present invention.

[0072] Figure 4 Schematic diagram of pseudo-color image of Indian Pines dataset of the present invention.

[0073] Figure 5 Schematic diagram of pseudo-color image of Pavia University dataset of the present invention.

[0074] Figure 6 It is a schematic diagram of the global perceptron module in the method of the present invention.

[0075] Figure 7 It is a schematic diagram of the partition sensor module in the method of the present invention.

[0076] Figure 8 It is a schematic diagram of the local sensor module in the method of the present invention.

[0077] Fig. 9 It is the classification result diagram of the Pavia University data set of the present invention.

[0078] Fig.10 This is the classification result diagram of the Indian Pines dataset of the present invention. DETAILED DESCRIPTION

[0079] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings. Specific implementation method 1

[0081] A hyperspectral image classification method based on reparameterized MLP in this specific implementation manner, the flow chart and the overall block diagram are respectively as follows Figure 1 and Figure 2 As shown, the following steps are included:

[0082] Step a: Input hyperspectral image

[0083] Input a hyperspectral image to be classified, and set each pixel point in the input hyperspectral image as a sample;

[0084] Step b: Obtain sample set

[0085] For the coordinate positions of all samples of the hyperspectral image, a coordinate transformation method is used to arrange the samples of each dimension of the hyperspectral image into a row vector to form a two-dimensional matrix, thereby obtaining a sample set of the spectral domain of the hyperspectral image; each column in the two-dimensional matrix is ​​a sample;

[0086] Where S is the number of pixels in a hyperspectral image with m rows and n columns, S = m × n, y sp represents the Sth pixel of the pth band; p represents the number of bands of the hyperspectral image;

[0087]

[0088] Step c: Determine the training sample set and the test sample set

[0089] Using the method of equal probability sampling, 10% of the samples in the sample set of the spectral domain of the hyperspectral image are randomly selected as the training sample set, and the coordinate positions of the training sample set of the spectral domain of the hyperspectral image are recorded; the remaining 90% of the samples are used as the test sample set of the spectral domain of the hyperspectral image;

[0090] Step d: Slicing of hyperspectral samples

[0091] Each pixel in the hyperspectral spectrum is processed as a fixed window to generate a single sample;

[0092] Step e: Build a reparameterized MLP model and train the reparameterized MLP model using training data

[0093] The re-parameterized MLP model includes a training phase and a testing phase; the training phase consists of three parts: a global perceptron module, a partitioned perceptron module, and a local perceptron module; and the re-parameterization is subsequently performed in the testing phase;

[0094] Step f: Classify the hyperspectral image

[0095] The hyperspectral feature map is fed into the Softmax function in the discriminator to finally complete the classification of the hyperspectral image and finally output the hyperspectral classification result map.

[0096] The idea of ​​the present invention is as follows:

[0097] Based on MLP, the present invention designs a more effective re-parameterized MLP model to improve the accuracy of hyperspectral image classification. The overall idea is that the global perceptron module branch 1 first segments the input hyperspectral feature map, and the branch 2 averages the original hyperspectral feature map, and then sends it to the BN layer and two fully connected layers; and broadcasts and adds it with the segmented hyperspectral feature map; the partitioned perceptron module is connected to the 1×1 group convolution and BN layer to transform the hyperspectral vector back to the original input shape; the local perceptron module sends the segmented hyperspectral feature map to the convolution layer; then re-parameterization is performed, and the convolution weights are fused to the fully connected layer; finally, the feature is input to the classification layer for classification.

[0098] The present invention introduces a ReLU function between two fully connected layers of a global perceptron, effectively avoiding the problems of gradient explosion and gradient vanishing; introduces a focal Loss to replace the cross entropy loss function, reduces the interference of a large number of simple samples, and focuses on identifying difficult samples; when using a gradient descent algorithm to optimize the objective function, as the global minimum of the Loss value is approached, the learning rate should become smaller to make the model as close to this point as possible, and the present invention introduces cosine annealing to reduce the learning rate through a cosine function; in the cosine function, as x increases, the cosine value first decreases slowly, then accelerates and then decreases slowly again; this descent mode can cooperate with the learning rate to produce a good effect in a very effective calculation method.

[0099] The MLP used in the present invention replaces the traditional convolutional layer because it has no convolution and self-attention mechanism and has achieved great success in the field of natural language processing. The multilayer perceptron (MLP) is also called the artificial neural network (ANN). In addition to the input and output layers, it can have multiple hidden layers in the middle. The simplest MLP contains only one hidden layer, that is, a three-layer structure, such as Figure 3 shown.

[0100] Assuming there are m samples and n features, the input layer x∈R m×n , assuming that the multilayer perceptron has only one hidden layer and that the hidden layer has h neurons, the weight and bias of the hidden layer can be expressed as W h ∈R n×h , b h ∈R 1×h ; There are q output label values, so the weight and bias parameters of the output layer are W 0 ∈R h×q , b h ∈R 1×q ; Then the input of the hidden layer and the output of the output layer can be expressed by H = XW h +b h calculate.

[0101] The input of the latter layer is the output of the previous layer. The above formula only performs a linear transformation on the data. In order to achieve the classification effect, an activation function must be introduced to perform a nonlinear transformation on each layer (except the input layer). The ReLU function is selected in the present invention. Specific implementation method 2

[0103] A hyperspectral image classification method based on a reparameterized MLP in this specific implementation is implemented on the basis of the first specific implementation and takes the hyperspectral image Indian Pines dataset as an example for classification. The Indian Pines dataset has a spatial resolution of 145×145, a band number of 200, and contains 16 labeled classes.

[0104] Step a: Input hyperspectral image

[0105] Input a hyperspectral image to be classified, and set each pixel point in the input hyperspectral image as a sample;

[0106] Step b: Obtain sample set

[0107] For the coordinate positions of all samples of the hyperspectral image, a coordinate transformation method is used to arrange the samples of each dimension of the hyperspectral image into a row vector to form a two-dimensional matrix, thereby obtaining a sample set of the spectral domain of the hyperspectral image; each column in the two-dimensional matrix is ​​a sample;

[0108] Where S is the number of pixels in a hyperspectral image with m rows and n columns, S = m × n, y sp represents the Sth pixel of the pth band; p represents the number of bands of the hyperspectral image;

[0109]

[0110] Step c: Determine the training sample set and the test sample set

[0111] Using the method of equal probability sampling, 10% of the samples in the sample set of the spectral domain of the hyperspectral image are randomly selected as the training sample set, and the coordinate positions of the training sample set of the spectral domain of the hyperspectral image are recorded; the remaining 90% of the samples are used as the test sample set of the spectral domain of the hyperspectral image;

[0112] Step d: Slicing of hyperspectral samples

[0113] Each pixel in the hyperspectral spectrum is processed as a fixed window to generate a single sample;

[0114] In this specific implementation, the Indian Pines dataset is a hyperspectral image from the Indian Pines Agricultural Experiment Station in northwest Indiana acquired using the Visible and Infrared Airborne Imaging Spectrometer (AVIRIS). It consists of 145×145 pixels and 200 spectral bands, including 16 classes; accordingly, Figure 4 The pseudo-color image of the scene and the ground reference data of the hyperspectral image dataset are given. The Pavia University dataset was obtained in northern Italy in 2002. It consists of 610×340 pixels and 115 spectral bands, including 9 classes. This example analyzes 103 spectral bands after removing 12 noise bands; accordingly, Figure 5 The pseudo-color image of the scene and the ground reference data of the hyperspectral image dataset are given. 10% of the samples are randomly selected as the training sample set of the spectral domain of the hyperspectral image, and the remaining 90% of the samples are used as the test sample set of the spectral domain of the hyperspectral image.

[0115] Step e: Build a reparameterized MLP model and train the reparameterized MLP model using training data

[0116] The re-parameterized MLP model includes a training phase and a testing phase; the training phase consists of three parts: a global perceptron module, a partitioned perceptron module, and a local perceptron module; and the re-parameterization is subsequently performed in the testing phase;

[0117] Step e1, the global perceptron module is as follows Figure 6 As shown in the figure, it includes two branches. Branch 1 first divides the input hyperspectral feature map, and branch 2 averages the original feature map, then sends the average pooled hyperspectral feature map to the BN layer plus two fully connected layers, and then adds it to the hyperspectral feature map obtained after the branch is divided; after completing the global perceptron operation, the obtained features are sent to the subsequent partition perceptron and local perceptron as the input of the partition perceptron and the local perceptron; specifically:

[0118] The global perceptron module includes two branches. Branch 1 first divides the input hyperspectral feature map to achieve parameter sharing of different partition features. After division, the hyperspectral feature map changes from 9×9×200 to 3×3×200. However, simple division will destroy the correlation between different partitions of the same channel. In order to make each partition relevant, the global perceptron adds branch 2 to process the hyperspectral features.

[0119] Branch 2 pairs of original features Fig. 9 ×9×200 for average pooling, the size of the hyperspectral feature map becomes 3×3×200, and the tensor R C×H×W represents the hyperspectral feature map, where C represents the number of channels, H and W represent the height and width of the input hyperspectral feature map, respectively; h and w represent the height and width of the hyperspectral feature map after segmentation, respectively; after calculating h 1 =3,w 1 =3, the calculation formula is h 1 =H / h and w 1 =W / w;

[0120] Then the hyperspectral features Figure 3 ×3×200 is fed into the BN layer and two fully connected layers. For the fully connected layers, P and Q represent the input and output dimensions respectively, V (in) and V (out) Represent the input and output of the fully connected layer network respectively; the fully connected kernel W∈R Q×P ;

[0121] Matrix multiplication (MMUL) is the output V of the fully connected layer (out) The calculation formula is:

[0122] V (out) =MMUL(V (in) ,W)=V (in) ·W T

[0123] After the BN layer and two fully connected layers, the hyperspectral vector is transformed into 1×1×200; then it is added to the hyperspectral feature map obtained after branch segmentation; after completing the global perceptron operation, the obtained features are sent to the subsequent partition perceptron and local perceptron as the input of the partition perceptron and local perceptron.

[0124] Step e2, partition sensor module as follows Figure 7 As shown, it contains a BN layer and a group convolution; specifically:

[0125] The partitioned perceptron module contains a BN layer and a group convolution; the input of the partitioned perceptron is 3×3×200, and then connected to a 1×1, groups=4 group convolution and BN layer; after the BN layer and a group convolution, the input 3×3×200 becomes the original hyperspectral feature input 9×9×200;

[0126] in represents the output feature; p is the number of filled pixels; F∈R C / g×K×K is the convolution kernel; g represents the number of convolution groups; M (out) The calculation formula is: (out) =gCONV(M (in) ,F,g,p)

[0127] Among them, the group convolution writes the group matrix multiplication (gMMUL) as:

[0128] M'=RS(V (in) ,(N,p,1,1)),F'=RS(W,p / g,l,l)

[0129] gCONV(V (in) ,W,g)=RS(gCONV(M',F',g,O),(N,Q)).

[0130] Step e3, the local sensor module is as follows Figure 8 As shown in the figure, there are convolution layers and BN layers with convolution kernel sizes of 1, 3, 5, and 7. In order to reduce the number of parameters, group convolution is used. The number of groups is consistent with that of the partitioned perceptron. First, the local perceptron module converts the segmented hyperspectral features into Figure 3 ×3×200 are simultaneously sent to the convolutional layers with kernel sizes of 1, 3, 5, and 7; for the subsequent hyperspectral feature addition operation, the convolutional layers need to be edge-filled; padding is to increase the number of pixels on each edge in order to keep the feature map not too small; then the feature map is sent to the BN layer, and the output of all convolutional branches is added to the output of the partitioned perceptron 3×3×200 as the final output;

[0131] During the training phase, RepMLP adds the output of the convolution to the output of the FC. This has the following benefits:

[0132] First, reduce FLOPs and increase speed. After converting all convolutions to equivalent FC kernels, due to the additivity of matrix multiplication AX+BX=(A+B)X, a sparse and shared parameter FC plus a non-sparse and non-shared parameter FC can be equivalently converted to one FC. The parameter of the equivalent conversion FC is the sum of the two added matrices. In this way, these convolutions can be equivalently removed. This idea also belongs to structural reparameterization.

[0133] Second, with the same number of parameters, the FLOPs of FC is much lower than that of traditional convolution.

[0134] Step e4, in the test phase, reparameterization is then performed, which is a process of fusing the convolution weights into the fully connected layer; in the test phase, the local perceptron module and the partitioned perceptron module are fused into a fully connected layer;

[0135] Since matrix multiplication is additive, the addition of the fully connected layer of the partitioned perceptron module and the convolutional layer of the local perceptron module is written as: MMUL(M (in) ,W (1) )+MMUL(M (in) ,W (2) )=MMUL(M (in) ,W (1) +W (2) )

[0136] Convolution is a sparse matrix multiplication, so there is an equivalent weight transformation, such as:

[0137] MMUL(M (in) ,W (1) )+MMUL(M (in) ,W (F,p) )=CONV(M (in) ,F,p)

[0138] Because in matrix multiplication, if you insert an identity matrix into it, it will not change the matrix result, so we have:

[0139] V out =V in *(I,W (F,p) )

[0140] Where I represents the identity matrix;

[0141] I=W (F,p)

[0142] It can be regarded as the result of convolution kernel F on the unit matrix;

[0143] For any input M (in) , convolution kernel F, there exists an FC kernel that satisfies:

[0144] M (out) =CONV(M (in) ,F,p)=MMUL(M (in) ,W (F,p) )

[0145] Using matrix multiplication, we have:

[0146] V out =V in ·W (F,p)τ

[0147] Insert the identity matrix I(Chw,Chw)) into it and apply the following law:

[0148] V out =V in ·(I·W (F,p)τ )

[0149] Among them, W (F,p) is constructed from F, I·W (F,p)τ Represents the feature M (I) Perform convolution, then we have:

[0150] M (I) =CONV(I,(Chw,C,h,w))

[0151] I.W. (F,p)τ =CONV(M (in) ,F,p)

[0152] V out =V in ·(I*W (F,p)τ ,(Chw,Ohw))

[0153] By comparing the above formula, we can get:

[0154] W (F,p) =CONV(M (I) ,F,p),(Chw,Ohw)

[0155] The above formula shows how to construct W from F and p (F,p) ; In short, the equivalent FC kernel of the convolution kernel can be obtained by convolving the identity matrix.

[0156] Step f: Classify the hyperspectral image

[0157] The hyperspectral feature map is fed into the Softmax function in the discriminator to finally complete the classification of the hyperspectral image and finally output the hyperspectral classification result map, such as Fig. 9 and Fig.10 shown.

[0158] The present invention replaces the traditional convolutional layer with an MLP structure, and proposes a modular reparameterized MLP model for image classification composed of multiple fully connected layers. The fully connected layer can efficiently model long-distance dependencies and position patterns, but cannot capture local information well. The present invention introduces convolution operations in the reparameterized MLP model to capture local information, and integrates the convolution kernel weights into the fully connected layer during the test phase. The model can make full use of the global representation capabilities of the fully connected layer and the local capture characteristics of the convolutional layer, and it has fewer parameters, making the classification model more flexible and efficient; further, the present invention replaces part of the convolutional layer with the fully connected layer to provide global representation capabilities and position perception capabilities. And the convolutional layer is introduced to give the fully connected layer the ability to capture local information that it does not have. Finally, through the reparameterization method, the convolutional layer and the fully connected layer are reparameterized into a fully connected layer to improve the reasoning speed.

[0159] The entire model is iterated and trained 200 times, and one iteration refers to traversing all the training data once. All experimental results are the average values ​​of 10 experiments. Epoch is set to 200, and the learning rate is set to 0.01. Finally, the comparison results of the classification of Pavia University and Indian Pines data sets using six different methods, RBF-SVM, EMP-SVM, CNN, SSRN, ResNet, and PyResNet, are given. The classification accuracy of various types of objects corresponding to each method is shown in Tables 1 and 3. The statistical results of OA (overall classification accuracy), AA (average classification accuracy) and Kappa coefficient of each method for the classification results of various types of objects are shown in Tables 2 and 4. It can be seen from the results in the table that compared with the deep learning model commonly used for hyperspectral classification, the present invention can obtain higher classification accuracy, and the accuracy is significantly improved. The method of the present invention is superior to all other models as a whole.

[0160] Table 1 Classification results of different classification methods of Pavia University dataset

[0161]

[0162] Table 2 Classification results of Pavia University dataset by different classification methods: OA, AA and Kappa statistics

[0163] RBF-SVM EMP-SVM CNN SSRN ResNet PyResNet Method of the present invention Overall accuracy OA 90.66±1.43 91.07±0.52 91.86±3.08 93.82±2.01 96.55±1.78 96.85±1.35 97.30±1.34 Average precision (AA) 87.25±0.49 88.12±2.23 88.27±2.01 94.61±1.56 95.62±1.68 96.28±1.65 96.98±1.56 100K 89.92±046 88.70±1.14 89.43±1.09 92.07±1.43 95.31±2.41 97.24±0.64 96.67±0.38

[0164] Table 3 Classification results of different classification methods for the Indian Pines dataset

[0165]

[0166] Table 4 Classification results of different classification methods for Indian Pines dataset OA, AA and Kappa statistics

[0167] RBF-SVM EMP-SVM CNN SSRN ResNet PyResNet Method of the present invention Overall accuracy OA 90.55±2.21 92.07±0.52 92.85±1.04 91.82±2.02 95.55±2.75 96.53±2.15 97.10±2.12 Average precision (AA) 87.25±0.23 88.22±2.21 88.27±2.02 92.52±2.56 95.62±2.63 96.27±2.65 97.65±2.56 100K 89.42±021 88.70±2.22 89.21±2.01 92.07±2.21 95.12±2.22 97.22±0.62 96.34±0.17

Claims

1. A hyperspectral image classification method based on reparameterized MLP, characterized in that: The following steps are involved: Step a: Input hyperspectral image Input a hyperspectral image to be classified, and set each pixel in the input hyperspectral image as a sample; Step b: Obtain sample set For the coordinate positions of all samples of the hyperspectral image, a coordinate transformation method is used to arrange the samples of each dimension of the hyperspectral image into a row vector to form a two-dimensional matrix, thereby obtaining a sample set of the spectral domain of the hyperspectral image; each column in the two-dimensional matrix is ​​a sample; Where S is the number of pixels in a hyperspectral image with m rows and n columns, S = m × n, y sp represents the Sth pixel of the pth band; p represents the number of bands of the hyperspectral image; Step c: Determine the training sample set and the test sample set Using the method of equal probability sampling, 10% of the samples in the sample set of the spectral domain of the hyperspectral image are randomly selected as the training sample set, and the coordinate positions of the training sample set of the spectral domain of the hyperspectral image are recorded; the remaining 90% of the samples are used as the test sample set of the spectral domain of the hyperspectral image; Step d: Slicing of hyperspectral samples Each pixel in the hyperspectral spectrum is processed as a fixed window to generate a single sample; Step e: Construct a reparameterized MLP model and train the reparameterized MLP model using the training data. The reparameterized MLP model includes a training phase and a testing phase. The training phase consists of a global perceptron module, a partitioned perceptron module, and a local perceptron module. In the testing phase, the convolution kernel weights of the convolution layer are converted into fully connected layer weights through a reparameterization operation, thereby simplifying the calculation and improving the inference efficiency. Step e1: The global perceptron module includes two branches. Branch 1 first divides the input hyperspectral feature map, and branch 2 averages the original feature map, then sends the average pooled hyperspectral feature map to the BN layer plus two fully connected layers, and then adds it to the hyperspectral feature map obtained after the branch is divided; after completing the global perceptron operation, the obtained features are sent to the subsequent partitioned perceptron and local perceptron as the input of the partitioned perceptron and the local perceptron; Step e2: The partitioned perceptron module contains a BN layer and a group convolution; Step e3: The local perceptron module contains convolution layers and BN layers with kernel sizes of 1, 3, 5, and 7. In order to reduce the number of parameters, grouped convolution is used, and the number of groups is consistent with that of the partitioned perceptron. First, the local perceptron module simultaneously sends the segmented hyperspectral feature map to the convolution layers with kernel sizes of 1, 3, 5, and 7. For the subsequent hyperspectral feature addition operation, the convolution layers need to be edge-filled. Padding is to increase the number of pixels on each edge. Then the feature map is sent to the BN layer, and the outputs of all convolution branches are added to the output of the partitioned perceptron as the final output. Step e4: Reparameterization is then performed during the test phase, specifically by fusing the convolutional weights of the convolutional layer into the fully connected layer, thereby simplifying the computation and improving the efficiency of inference. Specifically, the two parts of the local perceptron module and the partitioned perceptron module are fused into a fully connected layer, and the convolutional weights are fused with the fully connected layer weights through matrix multiplication to form a unified fully connected layer. This conversion process reduces computational complexity and enables the model to execute more efficiently during the inference phase; Step f: Classify the hyperspectral image The hyperspectral feature map is fed into the Softmax function in the discriminator to finally complete the classification of the hyperspectral image and finally output the hyperspectral classification result map.

2. A hyperspectral image classification method based on reparameterized MLP according to claim 1, characterized in that: Step e1 is specifically as follows: The global perceptron module consists of two branches. The first branch first divides the input hyperspectral feature map to achieve parameter sharing of different partition features. After division, the hyperspectral feature map changes from 9×9×200 to 3×3×200. However, simple division will destroy the correlation between different partitions of the same channel. In order to make each partition have correlation; Therefore, the global perceptron adds branch 2 to process the hyperspectral features; Branch 2 performs average pooling on the original feature map 9×9×200, and the size of the hyperspectral feature map becomes 3×3×200. C×H×W represents a hyperspectral feature map, where C represents the number of channels H, and W represents the height and width of the input hyperspectral feature map respectively; h, w are the height and width of the hyperspectral feature map after segmentation; after calculation, h1=3, w1=3, the calculation formula is h1=H / h, w1=W / w; Then the hyperspectral feature map 3×3×200 is sent to the BN layer and two fully connected layers. For the fully connected layer, P and Q represent the input and output dimensions respectively, and V (in) ,V (out) Represent the input and output of the fully connected layer network respectively; the fully connected kernel W∈R Q×P ; Matrix multiplication (MMUL) is the output V of the fully connected layer (out) The calculation formula is: V (out) =MMUL(V (in) ,W)=V (in) ·IN T The hyperspectral vector is transformed into 1×1×200 after the BN layer and two fully connected layers; then it is added to the hyperspectral feature map obtained after the branch is split; After completing the global perceptron operation, the obtained features are sent to the subsequent partition perceptron and local perceptron as the input of the partition perceptron and local perceptron.

3. The hyperspectral image classification method based on reparameterized MLP according to claim 1, characterized in that: Step e2 is specifically as follows: The partitioned perceptron module contains a BN layer and a group convolution. The input of the partitioned perceptron is 3×3×200, which is then connected to a 1×1, groups=4 group convolution and a BN layer. After the BN layer and a group convolution, the input 3×3×200 becomes the original hyperspectral feature input 9×9×200. in represents the output feature; p is the number of filled pixels; F∈R C / g×K×K is the convolution kernel; g represents the number of convolution groups; M (out) The calculation formula is: (out) =gCONV(M (in) ,F,g,p) The group convolution writes the group matrix multiplication (gMMUL) as: M′=RS(V (in) ,(N,p,1.1)),F′=RS(W,p / g,1.1) gCONV(V (in) ,W,g)=RS(gCONV(M′,F′,O),N,Q)。 4. The hyperspectral image classification method based on reparameterized MLP according to claim 1, characterized in that: Step e3 is specifically as follows: The local perceptron module contains convolutional layers and BN layers with kernel sizes of 1, 3, 5, and 7. In order to reduce the number of parameters, grouped convolution is also used here. The number of groups here is consistent with that of the partitioned perceptron. First, the local perceptron module simultaneously sends the segmented hyperspectral feature map 3×3×200 to the convolutional layers with kernel sizes of 1, 3, 5, and 7. It should be noted that for the subsequent hyperspectral feature addition operation, the convolutional layers here need to be edge-filled. Padding is to increase the number of pixels on each edge in order to keep the feature map not too small. The feature map is then sent to the BN layer, and the output of all convolution branches is added to the output of the partitioned perceptron 3×3×200 as the final output; During the training phase, RepMLP adds the output of the convolution to the output of the FC. This has the following benefits: First, reduce FLOPs and increase speed. After converting all convolutions into equivalent FC kernels, due to the additivity of matrix multiplication (AX+BX=(A+B)X), a sparse and shared parameter FC plus a non-sparse and non-shared parameter FC can be equivalently converted into a FC (whose parameters are the sum of these two matrices). In this way, these convolutions can be equivalently removed. This idea also belongs to structural reparameterization. Second, with the same number of parameters, the FLOPs of FC is much lower than that of traditional convolution.

5. The hyperspectral image classification method based on reparameterized MLP according to claim 1, characterized in that: Step e4 is specifically as follows: During the test phase, reparameterization is then performed, which is a process of fusing the convolutional weights into the fully connected layer; during the test phase, the local perceptron module and the partitioned perceptron module are fused into a fully connected layer; Since matrix multiplication is additive, the addition of the fully connected layer of the partitioned perceptron module and the convolutional layer of the local perceptron module is written as: MMUL(M (in) ,W (1) )+MMUL(M (in) ,W (2) )=MMUL(M (in) ,W (1) +W (2) ) Convolution is a sparse matrix multiplication, so there is an equivalent weight transformation, such as: MMUL(M (in) ,W (1) )+MMUL(M (in) ,W (F,p) )=CONV(M (in) ,F,p) Because in matrix multiplication, if you insert an identity matrix, it will not change the result of the matrix; so we have: V out =V in *(I,W (F,p) ) Where I represents the identity matrix; I=W (F,p) It can be regarded as the result of convolution kernel F on the unit matrix; For any input M (in) , convolution kernel F, there exists an FC kernel that satisfies: M (out) =CONV(M (in) ,F,p)=MMUL(M (in) ,W (F,p) ) Using matrix multiplication, we have: V out =V in ·W (F,p)τ Insert the identity matrix I(Chw,Chw) into it and apply the following law: V out =V in ·(I·W (F,p)τ ) Among them, W (F,p) is constructed from F, I·W (F,p)τ Represents the feature M (I) Perform convolution, then we have: M (I) =CONV(I,(Chw,C,h,w)) I·W (F,p)τ =CONV(M (in) ,F,p) V out =V in ·(I*W (F,p)T ,(Wow,Wow)) By comparing the above formula, we can get: W (F,p) =CONV(M (I) ,F,p),(Chw,Ohw) The above formula shows how to construct W through F and P (F,p) ; In short, the equivalent FC kernel of the convolution kernel can be obtained by convolving the identity matrix.

6. The hyperspectral image classification method based on reparameterized MLP according to claim 1, characterized in that: The Softmax function described in step f predicts the class label y∈{1,2…,15,16} of the input sample.

Images