Hyperspectral image classification method based on multi-scale involutional network

Abstract

The present invention discloses a method for hyperspectral image classification based on a scaled involution network, the method comprising: performing maximum and minimum normalization on the original hyperspectral data; constructing an adaptive spectral denoising network to purify the normalized data; constructing a spatial convolution subnetwork and an involution subnetwork, and embedding a Dropblock module in the network to improve the generalization ability of the network; constructing an autoencoder module, and applying regularization to the reconstruction loss to maintain the original characteristics of the image; fusing the spatial convolution subnetwork and the involution subnetwork, and simultaneously training the autoencoder module to finally obtain a classification and recognition result. The involution network constructed by the present invention pays more attention to the spectral characteristics of the hyperspectral data, and fuses the spatial convolution network to improve the spatial feature extraction ability of the model. At the same time, the introduced Dropblock module further improves the generalization of the model, and the autoencoder module can maintain the quality of the reconstructed image.

Description

Technical Field

[0001] The present invention relates to a hyperspectral image classification technology, and in particular to a hyperspectral image classification method based on a multi-scale involution network. Technical Background

[0002] Hyperspectral remote sensing technology uses a spectral imager to emit a large number of narrow-band electromagnetic waves at a target object to obtain relevant data. The spectrometer provides hundreds or even thousands of narrow-band spectral information for each pixel, allowing information that would otherwise be difficult to detect in a wide-band image to be mined. This allows for the integrated consideration of the target object's spatial structure and spectral information, enabling accurate description and prediction of the target object. Hyperspectral remote sensing technology has made significant contributions to related fields such as national defense science and technology, urban planning, and mineral exploration.

[0003] Hyperspectral image classification, a branch of hyperspectral image research, aims to assign an appropriate object label to each spectral pixel. Traditional algorithms for hyperspectral image classification include K-Nearest Neighbor (K-NN), Extreme Learning Machine (ELM), and Support Vector Machine (SVM). While these algorithms have some effectiveness for hyperspectral image classification, their overall performance is suboptimal.

[0004] In recent years, scholars have tended to introduce deep learning methods into the classification of hyperspectral images, and have achieved remarkable results. Due to its powerful feature extraction capabilities in the spatial dimension, many scholars have adopted convolutional neural networks (CNNs) to address many difficult problems in computer vision and have achieved great breakthroughs. Makantasis et al. designed a 2D convolutional neural network model that packs intermediate pixels into fixed-size blocks by filling in surrounding pixels, inputs them into the neural network for spatial information extraction, and finally sends them to a multi-layer perceptron for classification. However, more and more studies have found that this method of simply using the spatial information of hyperspectral images is difficult to achieve the desired effect. Therefore, researchers have focused more on methods that combine the spatial and spectral information of hyperspectral images. Summary of the Invention

[0005] The present invention discloses a method for hyperspectral image classification based on a multi-scale involutional network. The involutional network focuses on the spectral information of the hyperspectral image, and at the same time integrates the spatial convolutional network to associate the spatial-spectral features, and separately sets an autoencoder to ensure the quality of the reconstructed image.

[0006] The technical solution to achieve the present invention is: a method for hyperspectral image classification based on a multi-scale involutional network, comprising the following steps:

[0007] (1) Perform maximum and minimum normalization on the original hyperspectral image dataset;

[0008] The normalized hyperspectral image is denoted as I, and the size of I is H×W×C, where H and W represent the length and width of the hyperspectral image, C is the number of spectra of the hyperspectral image, and the number of categories of the hyperspectral image is denoted as M.

[0009] (2) Construct an adaptive spectral denoising network, that is, through several batches of normalization layers, convolution layers, and pooling layers, the number of feature maps in the last layer is made smaller than the number of channels of the normalized hyperspectral image, and the spectral information that is effective for the current task is adaptively learned;

[0010] Among them, the feature map after the denoising network is recorded as I1, the size is H×W×C1, C1<C;

[0011] (3) Construct a multi-scale inner volume sub-network, that is, construct different scale neighborhood spaces for each spectral vector in I1 Where s ≥ 1; then compress the spectral vector through a small 1×1 convolution kernel, and the compressed spectral vector is recorded as T, where the channel size of T is consistent with the neighborhood space, which is s×s; expand T to the plane along the channel direction, that is, obtain the involution kernel of multiple scales; multiply the involution kernel with the neighborhood space of the spectral vector, and then sum the regional information to obtain the spectrum-related features, and finally fuse the output features of multiple scales, recorded as I 11 ;

[0012] (4) Construct a spatial convolution subnetwork, that is, by using multiple three-dimensional convolution kernels to extract the deep spatial information of the feature map I1, and embed the Dropblock block in the network to discard the K×K×R size neighborhood spatial pixels in the feature map with probability p; the feature map obtained after I1 passes through the spatial convolution subnetwork is denoted as I 12 ;

[0013] (5) Fusion of spectral features extracted by multi-scale involution subnet I 11 , and the spatial features I extracted by the spatial convolution sub-network 12 The fused feature is recorded as I2. The Softmax function is used on I2 to convert the feature extraction result into a probability distribution. The cross entropy is used as the loss function of the fusion network to calculate the difference L1 between the prediction result and the object label.

[0014] (6) Construct an autoencoder, that is, use a three-dimensional convolution kernel to encode and decode the feature information of I1 to reconstruct the image. The reconstructed image is recorded as The mean square error is selected as the loss function of the autoencoder, I and The loss value between them is recorded as L2;

[0015] (7) Calculate the total loss, that is, L1 is used as the main loss to obtain refined spatial spectral features, and L2 is used as a regularization term to maintain the reconstruction quality of the image and prevent the intermediate features from deviating from the original distribution; the total loss is recorded as L, L = L1 + L2, which is used to train the network, and the trained network is used to label the samples to be classified.

[0016] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for hyperspectral image classification based on a multi-scale involution network is implemented.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for hyperspectral image classification based on a multi-scale involution network.

[0018] Compared with the existing technology, the present invention has the following significant advantages: (1) the convolution kernel is divided into multiple scales, which enhances the perception of spectral features of hyperspectral images at different scales, and at the same time integrates the spatial feature extraction capability of the convolutional network, so that the spatial spectrum characteristics of the hyperspectral image can be fully expressed; (2) the Dropblock module is introduced to further improve the generalization of the network; (3) the constructed autoencoder can effectively improve the quality of the reconstructed image; (4) the method of the present invention can be widely applied to the classification of small sample hyperspectral images.

[0019] The present invention is further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Flowchart of the method of the present invention.

[0021] Figure 2 This is a network structure diagram of the present invention.

[0022] Figure 3 Schematic diagram of the involution structure.

[0023] Figure 4 These are the IP-RGB renderings and real object distribution maps of the Indian pine dataset, where (a) is the IP-RGB rendering and (b) is the real object distribution map.

[0024] Figure 5These are the classification effect diagrams of the Indian pine dataset, where (a), (b), and (c) are the classification effect diagrams using the 2D-CNN method, the 3D-CNN method, and the method of the present invention, respectively. DETAILED DESCRIPTION

[0025] The present invention proposes a method for hyperspectral image classification based on a multi-scale convolutional network, comprising: performing maximum and minimum normalization on the original hyperspectral data; constructing an adaptive spectral denoising network to further purify the normalized data; constructing a spatial convolutional subnetwork and an involutional subnetwork based on the spatial structure and spectral characteristics of the hyperspectral data, respectively, and embedding a Dropblock module in the network to improve the network generalization ability; constructing an autoencoder module and applying regularization to the reconstruction loss to maintain the original image characteristics; fusing the spatial convolutional subnetwork and the multi-scale involutional subnetwork, and simultaneously training the autoencoder to ultimately obtain classification and recognition results. The advantage of the present invention is that the constructed involutional network focuses more on the spectral characteristics of the hyperspectral data, and fuses the spatial convolutional network to improve the spatial feature extraction ability of the model. At the same time, the introduced Dropblock module further improves the generalization of the model, while the autoencoder module can maintain the quality of the reconstructed image. This makes it possible to fully utilize the spatial-spectral characteristics of the hyperspectral image even when there are fewer training samples, and obtain higher classification accuracy.

[0026] The following is combined with Figure 1 , the steps of the present invention are described in detail.

[0027] Step 1: Perform maximum and minimum normalization on the original hyperspectral image;

[0028] According to the maximum and minimum values ​​in the hyperspectral image, the data is shrunk to the [0,1] interval. The normalization formula is as follows:

[0029]

[0030] Among them, I min and I max Represent the maximum and minimum values ​​in the hyperspectral image respectively. The size of the normalized hyperspectral image is H×W×C, where H, W, and C represent the height, width, and number of spectra of the hyperspectral image respectively. The normalized image is fed into the network. The network structure of the present invention is as follows: Figure 2 shown.

[0031] Step 2: Build an adaptive denoising network.

[0032] A two-layer network was constructed with a batch normalization (BN) layer, a convolutional layer, and a LeakyRelu layer; a batch normalization (BN) layer, a convolutional layer, and a LeakyRelu layer. A 2D convolution kernel with a size of 1×1 and channels of 128 and 64 was used. The convolution stride was set to 1.

[0033] The feature map of I after the adaptive denoising network is denoted as I1, and its size is H×W×64.

[0034] Step 3: Construct a multi-scale inward convolutional subnet and a spatial convolutional network.

[0035] The multi-scale involution subnet consists of an involution layer and a pooling layer, where the involution layer defines the involution operation. The involution operation is as follows: Figure 3 As shown, multiple scale neighborhood spaces are constructed for I1. Where s≥1. Take s=1, s=3, s=5 as three scales, and the corresponding neighborhood spaces are recorded as Z1, Z2, and Z3. Use a 1×1 convolution kernel to transform the spectral channel size of I1 to s 2 The transformed feature map T is expanded along the spectral dimension into an involution kernel of size s×s, which are denoted as K1, K2, and K3 respectively. The involution process can be expressed as:

[0036]

[0037] Among them, Invo is the output of I1 after the involution operation, It means multiplying the elements of the corresponding positions of two matrices, and sum(*) means summing along the channel dimension.

[0038] After I1 passes through multiple convolution layers and pooling layers, the output feature map is I 11 .

[0039] The inner convolution subnet consists of multiple convolutional layers, pooling layers, and Dropblock layers.

[0040] Among them, the convolution layer uses 3D convolution kernels, the sizes of the convolution kernels are 5×5×11, 1×1×5, and 1×1×3, the step sizes are set to 1×1×4, 1×1×2, and 1×1×2, the padding sizes are 2×2×5, 0×0×2, and 0×0×1, and the output feature map sizes are 10, 10, and 2.

[0041] The pooling layer uses the LeakyRelu() function.

[0042] The drop probability of the Dropblock layer is set to 0.1, and the size of the dropped area is 3×3×3.

[0043] After I1 passes through multiple convolutional layers, pooling layers, and Dropblock layers, the output feature map is I 12 .

[0044] Fusion I1 and I2 are used as the final feature map. The fusion process is:

[0045]

[0046] in Represents the fusion operation, let I 11 The number of channels is r 11 , I 12 The number of channels is r 12 , then the number of channels of the fused feature I2 is r 11 +r 12 At the same time, the Softmax function is used to convert the value of I2 into a probability distribution. The normalized I2 is expressed as:

[0047]

[0048] Among them, I′2 represents the feature map after I2 normalization, Represents the i-th point in the feature map I2, N represents the total number of pixels in I2, and ∑ represents the summation operation. Finally, the cross entropy loss function is used to calculate the loss L1 between the true label and the target. The calculation process of L1 is:

[0049]

[0050] Where F represents the number of categories of hyperspectral images, y k is the unique label representation of the sample label. When the sample belongs to the kth category, y k =1, otherwise y k =0.

[0051] Step 4: Build the autoencoder.

[0052] The autoencoder consists of a convolutional layer with a 3D convolution kernel of 1×1×25 size, a step size of 1×1×1, and an output feature map size of 200, denoted as The mean square error is used to calculate the reconstruction loss L2 of the original image. The calculation process of the reconstruction loss is:

[0053]

[0054] Step 5: Calculate the total loss and use it to train the network.

[0055] The total loss is recorded as L, which is the linear combination of L1 and L2. The calculation process of L is:

[0056] L=λL1+(1-λ)L2

[0057] Here, λ is a hyperparameter in the range [0,1] that controls the contribution ratio of L1 and L2 losses. Finally, the network model is trained using L, and the trained model is used to label the samples to be classified.

[0058] Compared to existing methods, this paper addresses the more difficult-to-process spectral information in hyperspectral images by designing a multi-scale involutional network to extract spectral features and combining it with a spatial convolutional network to supplement the spatial information. This method fully utilizes the spatial spectral characteristics of hyperspectral images and demonstrates superior performance in small-sample hyperspectral image classification.

[0059] The effect of the present invention can be further illustrated by the following simulation experiments:

[0060] (1) Simulation conditions

[0061] The hyperspectral data used in the simulation of the present invention is the Indian Pines dataset.

[0062] The present invention selects the Indian Pines data set as the experimental data used for simulation. Its pseudo-color map and GroudTruth map are as follows Figure 4 The Indian Pines dataset is a collection of hyperspectral images collected by the AVIRIS sensor at the Indian Pines Experimental Site in western Indiana. The dataset consists of 145×145 pixels and 200 spectral reflectance bands, encompassing 16 ground feature types. The images are in mat2d format, and 5% of the training samples were selected for experiments. Simulations were conducted using Python 3.6 on Windows 10.

[0063] In order to quantify the classification results, this experiment uses the following three evaluation indicators:

[0064] (1) Overall accuracy OA: The number of correctly classified pixels in the test set divided by the total number of pixels is called overall accuracy OA. Its distribution range is between 0 and 100%. The larger the value, the better the classification effect.

[0065] (2) Average accuracy AA: divide the number of correctly classified pixels in each category on the test set by the total number of all pixels in that category to obtain the correct classification accuracy of that category. The average of the accuracy of all categories is called average accuracy AA, which ranges from 0 to 100%. The larger the value, the better the classification effect.

[0066] (3) Kappa (Kappa Coefficient) coefficient: The Kappa coefficient is an evaluation indicator defined on the confusion matrix X. It comprehensively considers the elements on the diagonal of the confusion matrix and the elements off the diagonal, and more objectively reflects the classification performance of the algorithm. The distribution range of Kappa is between 0 and 1. The larger the value, the better the classification effect.

[0067] (2) Simulation content

[0068] The simulation experiment of the present invention is simulated with two existing hyperspectral image classification methods, namely a hyperspectral image classification method based on a 2D convolutional network (2D-CNN) and a hyperspectral image classification method based on a 3D convolutional network (3D-CNN).

[0069] (3) Analysis of simulation experiment results

[0070] Table 1 and Figure 5 The corresponding classification results are shown.

[0071] Table 1

[0072]

[0073] The simulation experimental results of the above three sets of real data sets show that the present invention is superior to the traditional convolution algorithm in terms of OA, AA and Kappa coefficient because it integrates the spectral characteristics of hyperspectral images.

[0074] This invention can achieve ideal classification accuracy on small samples and can be widely used in the classification of small sample hyperspectral images.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for hyperspectral image classification based on a multi-scale involutional network, characterized in that: The following steps are involved: (1) Perform maximum and minimum normalization on the original hyperspectral image dataset; The normalized hyperspectral image is denoted as I, and the size of I is H×W×C, where H and W represent the length and width of the hyperspectral image, C is the number of spectra of the hyperspectral image, and the number of categories of the hyperspectral image is denoted as M. (2) Construct an adaptive spectral denoising network, that is, through several batches of normalization layers, convolution layers, and pooling layers, the number of feature maps in the last layer is made smaller than the number of channels of the normalized hyperspectral image, and the spectral information that is effective for the current task is adaptively learned; Among them, the feature map after the denoising network is recorded as I1, the size is H×W×C1, C1<C, C1 is the spectral number of the feature map after the denoising network; (3) Construct a multi-scale inner volume sub-network, that is, construct different scale neighborhood spaces for each spectral vector in I1 Where s ≥ 1; then compress the spectral vector through a small 1×1 convolution kernel, and the compressed spectral vector is recorded as T, where the channel size of T is consistent with the neighborhood space, which is s×s; expand T to the plane along the channel direction, that is, obtain the involution kernel of multiple scales; multiply the involution kernel with the neighborhood space of the spectral vector, and then sum the regional information to obtain the spectrum-related features, and finally fuse the output features of multiple scales, recorded as I 11 ; (4) Construct a spatial convolution subnetwork, that is, by using multiple three-dimensional convolution kernels to extract the deep spatial information of the feature map I1, and embed the Dropblock block in the network to discard the K×K×R size neighborhood spatial pixels in the feature map with probability p; the feature map obtained after I1 passes through the spatial convolution subnetwork is denoted as I 12 ; (5) Fusion of spectral features extracted by multi-scale involution subnet I 11 , and the spatial features I extracted by the spatial convolution sub-network 12 The fused feature is recorded as I2. The Softmax function is used on I2 to convert the feature extraction result into a probability distribution. The cross entropy is used as the loss function of the fusion network to calculate the difference L1 between the prediction result and the object label. (6) Construct an autoencoder, that is, use a three-dimensional convolution kernel to encode and decode the feature information of I1 to reconstruct the image. The reconstructed image is recorded as The mean square error is selected as the loss function of the autoencoder, I and The loss value between them is recorded as L2; (7) Calculate the total loss, that is, L1 is used as the main loss to obtain refined spatial spectral features, and L2 is used as a regularization term to maintain the reconstruction quality of the image and prevent the intermediate features from deviating from the original distribution; the total loss is recorded as L, L = L1 + L2, which is used to train the network, and the trained network is used to label the samples to be classified.

2. The method for hyperspectral image classification based on a multi-scale involutional network according to claim 1, characterized in that: In step (1), the original hyperspectral image dataset is normalized to the maximum and minimum values. According to the maximum and minimum values ​​in the hyperspectral image, the data is shrunk to the interval [0, 1]. The normalization formula is as follows: Among them, I min and I max represent the minimum and maximum values ​​in the hyperspectral image, respectively.

3. The method for hyperspectral image classification based on a multi-scale involutional network according to claim 1, characterized in that: The adaptive spectral denoising network described in step (2) is constructed, and multiple 1×1 convolution kernels are used to compress the spectral number of I to C1, C1<C.

4. The method for hyperspectral image classification based on a multi-scale involutional network according to claim 1, characterized in that: The multi-scale inner volume sub-network described in step (3) is constructed to construct the neighborhood space of multiple scales based on the denoised feature map I1 where s ≥ 1; take s = 1, s = 3, s = 5, then the neighborhood spaces of the three scales are recorded as Z1, Z2, Z3; use a 1×1 convolution kernel to transform the spectral channel size of I1 to s 2 The transformed feature map T is expanded along the spectral dimension into an involution kernel of size s×s. The involution kernels are denoted as K1, K2, and K3 respectively. The involution process can be expressed as: Among them, Invo is the output of I1 after the involution operation, Indicates the multiplication of the corresponding position elements of two matrices, and sum(*) indicates the sum along the channel dimension; After I1 passes through multiple convolution layers and pooling layers, the output feature map obtained by the multi-scale convolution subnet is denoted as I 11 .

5. The method for hyperspectral image classification based on a multi-scale involutional network according to claim 1, characterized in that: In step (4), a spatial convolution subnet is constructed, and multiple three-dimensional convolution kernels are used to further extract the deep spatial information of I1. Dropblock is introduced to regularize the convolution layer. The output feature map obtained after I1 passes through the spatial convolution subnet is denoted as I 12 .

6. The method for hyperspectral image classification based on a multi-scale involutional network according to claim 1, characterized in that: In step (5), the features extracted by the multi-scale inner convolution subnet and the spatial convolution subnet are fused. The fusion process is: in Represents the fusion operation, let I 11 The number of channels is r 11 , I 12 The number of channels is r 12 , then the number of channels of the fused feature I2 is r 11 +r 12 At the same time, the Softmax function is used to convert the value of I2 into a probability distribution. The normalized I2 is expressed as: Among them, I'2 represents the feature map after I2 normalization, Represents the i-th point in the feature map I2, N represents the total number of pixels in I2, and ∑ represents the summation operation; The cross entropy loss function is used to calculate the loss L1 between I'2 and the true label. The calculation process of L1 is: Where M represents the number of categories of hyperspectral images, y k is the one-hot encoding of the sample label. When the sample belongs to the kth category, y k =1, otherwise y k =0.

7. The method for hyperspectral image classification based on a multi-scale involutional network according to claim 6, characterized in that: In step (6), an autoencoder is constructed, and a 1×1 3D convolution kernel is used to encode and decode the feature information of I1 to reconstruct the image. The reconstructed image is recorded as The mean square error is selected as the loss function of the autoencoder, I and The loss value between is recorded as L2, and the calculation process of reconstruction loss is:

8. The method for hyperspectral image classification based on a multi-scale involutional network according to claim 1, characterized in that: In step (7), the total loss is calculated and recorded as L, where L is the linear combination of L1 and L2. The calculation process of L is: L=λL1+(1-λ)L2 Among them, λ is a hyperparameter in the range of [0,1], which is used to control the loss contribution ratio of L1 and L2; finally, L is used to train the network model, and the trained model is used to label the samples to be classified.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for hyperspectral image classification based on a multi-scale involution network as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for hyperspectral image classification based on a multi-scale involution network as described in any one of claims 1 to 8 is implemented.

Images