A hyperspectral remote sensing image classification method based on hybrid convolutional neural network

Abstract

The present invention discloses a method for classifying hyperspectral remote sensing images using a hybrid convolutional neural network. The present invention comprises: 1. connecting 3D and 2D convolutional neural networks in series, adding parallel convolution blocks and non-local attention modules, and constructing a custom neural network; 2. dividing annotated hyperspectral images into a number of overlapping small data cubes as inputs of the model; 3. dividing samples into training sets and test sets according to a certain ratio; 4. inputting training set samples into the neural network in batches for training until the model is stable; 5. using the trained model to classify the hyperspectral image, evaluating the classification effect of the model through the pre-divided test set, and finally obtaining a classification result graph. The present invention makes full use of the rich spectral information and spatial information of the hyperspectral remote sensing image. At the same time, 1×1 convolution is used to compress the channel dimension of the feature map during the attention operation, thereby reducing the amount of calculation of the model and improving the training speed of the model.

Description

Technical Field

[0001] The present invention relates to the field of graphics processing, and in particular to a hyperspectral remote sensing image classification method based on a hybrid convolutional neural network. Background Art

[0002] The band width of hyperspectral remote sensing images is usually about tens of nanometers, which is much narrower than the bandwidth of multispectral images. Therefore, hyperspectral remote sensing images have richer spectral information and are widely used in various fields. Hyperspectral remote sensing images have a natural advantage in classification tasks with rich spectral information. Therefore, hyperspectral remote sensing is widely used in precision agriculture, rock and mineral identification, environmental monitoring, marine remote sensing and other fields.

[0003] For the classification task of hyperspectral remote sensing images, traditional methods include classification methods based on spectral features and classification methods based on data statistical features, but they cannot fully utilize the spatial information and rich spectral information of hyperspectral remote sensing images, and the classification accuracy is often not high. In recent years, some deep learning models have been introduced into the field of hyperspectral remote sensing image classification, especially convolutional neural networks. Convolutional neural networks use convolutional layers to extract features of input data, and use pooling layers to downsample feature maps extracted from convolutional layers, expand the receptive field of convolution kernels, and compress data. Through the superposition of convolutional layers and pooling layers, convolutional neural networks can not only extract shallow texture features in images, but also extract deep semantic features.

[0004] However, due to the high redundancy of spectral information in hyperspectral images, using convolutional neural networks to classify them often requires a deeper network structure. At the same time, since the convolution kernels of the same layer use the same size, it is easy to have insufficient feature information extraction, which also leads to low classification accuracy. Summary of the invention

[0005] The present invention aims at the existing problems in hyperspectral remote sensing image classification and provides a hyperspectral remote sensing image classification method based on a hybrid convolutional neural network. The method constructs a hybrid convolutional neural network hyperspectral image classification model based on parallel convolution blocks and non-local attention mechanism, including a multi-scale feature extraction and fusion module, a spatial-spectral attention module, a spatial attention module and a classification module.

[0006] The multi-scale feature extraction and fusion module adopts a parallel convolution block structure, and the convolution layers of different branches in this module use convolution kernels of different sizes. A batch normalization layer and an activation layer are added after each convolution layer. The convolution operation is set to fill the feature map to keep the size of the feature map consistent. Figure Ⅰ Fusion is performed to obtain the features of fusion multi-scale features Figure II .

[0007] The spatial-spectral attention module uses a non-local attention mechanism to focus on features Figure III Perform attention operation, reduce the dimension of data through the convolution layer with a convolution kernel size of 1, and suppress features Figure III The irrelevant information in the feature map makes the effective information more prominent. When calculating the output of each pixel position in the feature map, the non-local attention mechanism no longer only calculates the neighboring pixels, but calculates the correlation of all position pixels in the feature map, and then uses the correlation as a weight to represent the similarity between other position pixels and the current pixel to be calculated. Since hyperspectral images have both spatial and spectral dimensions, non-local operations can be performed in both spatial and spectral dimensions. By calculating the correlation between pixels in the feature map, a more comprehensive feature map can be obtained. Figure IV .

[0008] Described features Figure III It is obtained through a 3D convolutional layer and used as input to the spatial-spectral attention module.

[0009] Furthermore, two convolutional layers + activation layers are connected in series after the spatial-spectral attention module to further extract the spatial-spectral feature map.

[0010] Furthermore, by using a 3D convolutional layer with a spectral dimension equal to the number of spatial-spectral feature maps, the depth is normalized and the features after the depth dimension are removed. Figure V Features extracted by the multi-scale feature extraction module Figure II The aggregation is used as the input of the spatial attention module.

[0011] The spatial attention module is similar to the spatial-spectral attention module. The spatial attention module also uses the non-local attention mechanism to perform attention operations on the feature map. It uses a 2D convolution kernel and performs non-local operations only in the spatial dimension to calculate the correlation of pixels at different spatial positions.

[0012] Furthermore, two layers of 2D convolutional layers + batch normalization layers and activation layers are added after the spatial attention module to further extract spatial feature maps.

[0013] Furthermore, the classification module includes a global average pooling layer, a fully connected layer and a softmax classification function. The global average pooling layer aggregates the spatial feature maps extracted by the model, compresses the data volume, and then serves as the input of the fully connected layer, and finally uses the softmax function for classification.

[0014] The method steps of hyperspectral image classification proposed by the present invention include:

[0015] (1) Model construction: Use 3D convolutional neural network, 2D convolutional neural network, parallel convolution blocks and non-local attention mechanism to build a neural network;

[0016] (2) Sample extraction: Extract the pixels to be classified and a certain number of neighboring pixels centered on them from the original hyperspectral remote sensing image as training samples and serve as the input of the network;

[0017] (3) Data division: Divide the training samples into training set and test set according to a certain ratio;

[0018] (4) Model training: Use the training set to train the model to obtain parameters suitable for hyperspectral remote sensing image classification;

[0019] (5) Classification: Use the trained network to classify the hyperspectral remote sensing image and obtain the classification results.

[0020] The beneficial effects of the present invention are as follows:

[0021] The present invention provides a method for classifying hyperspectral remote sensing images based on a hybrid convolutional neural network. The multi-scale feature extraction and fusion module uses convolution kernels of different sizes and a maximum pooling layer to perform convolution and pooling operations on the input feature map, extract spatial features of different scales, and then fuse them into feature maps with spatial features of different scales. The feature information of the obtained feature map at each scale is richer and more comprehensive, which is conducive to further extracting classification features in subsequent convolution operations, and further improving the classification accuracy of hyperspectral images.

[0022] The spatial-spectral attention module and the spatial attention module enhance the effective information of the feature map and suppress irrelevant information in the spatial and spectral dimensions, thereby realizing the joint learning of spatial-spectral features and making full use of the rich spectral and spatial information of hyperspectral remote sensing images. At the same time, 1×1 convolution is used to compress the channel dimension of the feature map during the attention operation, reducing the amount of model calculation and improving the training speed of the model.

[0023] The hyperspectral remote sensing image classification method based on hybrid convolutional neural network provided by the present invention is used to classify two public hyperspectral image datasets of Pavia University and Salinas, and the overall classification accuracy of 99.92% and 99.81% and the average classification accuracy of 99.89% and 99.88% are achieved respectively, which effectively improves the classification accuracy of hyperspectral images, and the obtained classification map is uniform and smooth, and the boundaries are clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solution used in the present invention, the following briefly introduces the drawings required for the technical description.

[0025] Figure 1 It is a schematic diagram of a flow chart of a network designed in an embodiment of the present invention;

[0026] Figure 2 Schematic diagram of the multi-scale feature extraction and fusion module used in the present invention;

[0027] Figure 3 is a schematic diagram of the attention module used in the present invention;

[0028] Figure 4 It is a schematic diagram of the hyperspectral remote sensing image of the present invention.

[0029] Figure 5 It is a schematic diagram of the hyperspectral remote sensing image of the present invention.

[0030] in, Figure 4 -a, Figure 5 -a is a true color composite image of the hyperspectral remote sensing image used in the experiment of the present invention.

[0031] Figure 4 -b, Figure 5 -b is the ground-truth map of the hyperspectral remote sensing image used in the experiment of the present invention.

[0032] Figure 4 -c, Figure 5 -c is a hyperspectral remote sensing image classification result diagram provided by an example of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0034] The invention discloses a method for classifying hyperspectral remote sensing images using a hybrid convolutional neural network, the method comprising: building a hybrid neural network of a 3D convolutional neural network and a 2D convolutional neural network in series; adding a parallel convolutional block structure to the hybrid neural network to provide receptive fields of different sizes for extracting and fusing multi-scale features; using a non-local attention mechanism to calculate the correlation between the spatial dimension and the spectral dimension of a pixel to obtain a more comprehensive feature map; using global average pooling to aggregate the information of the feature map, and combining a fully connected layer to achieve pixel-level hyperspectral remote sensing image classification. In summary, the invention combines a 3D convolutional neural network, a 2D convolutional neural network, a multi-scale feature extraction and fusion module, and a space and space-spectral attention module to build a neural network for hyperspectral remote sensing image classification. On the basis of extracting features using a convolutional neural network, a multi-scale feature extraction and fusion module based on a parallel convolutional block structure extracts multi-scale features in the feature map and fuses them. The space-spectral attention module and the space attention module calculate the correlation between pixels to further obtain a more complete and comprehensive feature map, thereby improving the classification accuracy of the network for hyperspectral remote sensing images.

[0035] The specific implementation steps of the present invention are as follows:

[0036] Step (1) connects the 3D convolutional neural network and the 2D convolutional neural network in series, adds parallel convolution blocks and non-local attention modules, and constructs a custom neural network;

[0037] Step (2) divides the labeled hyperspectral image into several overlapping H×W×C small data cubes as the input of the model;

[0038] Step (3) divide the samples into training set and test set according to a certain ratio;

[0039] Step (4) inputting the training set samples into the neural network in batches for training until the model is stable;

[0040] Step (5) uses the trained model to classify the hyperspectral image, evaluates the classification effect of the model through a pre-divided test set, and finally obtains the classification result graph.

[0041] Furthermore, the specific implementation process of step (1) is as follows:

[0042] Figure 1 The figure is a structural diagram of the neural network designed by the present invention. The neural network designed by the present invention can be divided into two branches. In one branch, the first layer of the network is a 3D convolution layer with a step size of 2, a convolution kernel size of 3×3×7, and a convolution kernel number of 24. The information of the hyperspectral image is initially aggregated through the 3D convolution layer to obtain the feature Figure III , reducing the amount of subsequent calculations;

[0043] Next is a spatial-spectral attention module that calculates the correlation of pixels in the spatial and spectral dimensions to obtain more comprehensive features. Figure IV Formula (1) is the general calculation formula of the non-local attention mechanism:

[0044]

[0045] Among them, y i Represents the output corresponding to the pixel at position i in the feature map, X i represents the pixel at position i in the input feature map, X j represents the pixels at all other positions in the feature map, and c(X) is the normalization function. The f function in formula (1) is used to calculate the similarity between pixels, which is implemented by embedding Gaussian function, as shown in formula (2):

[0046]

[0047] Among them, θ(X i )=W θ X i , W θ and is the weight matrix to be learned,

[0048] The g function is a linear transformation function, as shown in formula (3):

[0049] g(X j )=W g X j (3)

[0050] Among them, W g It is also the weight matrix to be learned.

[0051] Normalization function c(X), as shown in formula (4):

[0052]

[0053] After the spatial-spectral attention module, there are two layers of 3D convolutional layers with a convolution kernel size of 3×3×5 and a number of 24, as well as corresponding batch normalization layers and activation layers (the activation function used in the activation layer is the ReLU function) to further extract spatial-spectral information. Then there is a layer of 3D convolutional layers with a convolution kernel size of 1×1×K (K is the number of feature maps) and a number of 24 convolution kernels, as well as batch normalization layers and activation layers. The spectral dimension is normalized by using a convolution kernel scale with the same size as the spectral dimension of the spatial-spectral feature map to obtain the features after the spectral dimension is compressed. Figure V ;

[0054] The other branch is a multi-scale feature extraction fusion module, which uses a parallel convolution block structure and multiple 2D convolution kernels of different sizes to extract multi-scale spatial features of hyperspectral images. It provides three sizes of receptive fields: 1×1, 3×3, and 5×5. It also uses 1×1 convolution and 3×3 maximum pooling to aggregate information. Finally, the features extracted by the convolution kernels are combined. Figure Ⅰ Aggregate to obtain features that fuse multi-scale features Figure II ;

[0055] The features obtained from the above two branches Figure II and Features Figure V Perform concatenation and then input into the spatial attention module to further calculate the spatial correlation between pixels;

[0056] The spatial features are further extracted through two layers of 2D convolutional layers with a convolution kernel size of 3×3 and a number of 24 + batch normalization layers and activation layers. Finally, the obtained feature maps are globally average pooled and connected to the fully connected layer for classification.

[0057] The difference between the spatial-spectral attention module and the spatial attention module is whether the spectral dimension is involved in the calculation. The spatial attention module only calculates the spatial correlation of pixels, while the spatial-spectral attention module can calculate the correlation of pixels in both the spatial and spectral dimensions. Figure 2 This is a schematic diagram of the non-local attention mechanism.

[0058] The multi-scale feature extraction fusion module achieves different receptive fields by using convolution kernels of different sizes in the same layer of the network. The feature maps obtained by convolution kernels of different sizes are fused to obtain feature maps with multi-scale features, thus avoiding the problem of insufficient feature extraction at a single scale. Figure 3 It is a schematic diagram of the multi-scale feature extraction and fusion module.

[0059] Step (2) divides the labeled hyperspectral image into several overlapping H×W×C small data cubes as the input of the model;

[0060] Furthermore, the specific implementation process of step (2) is as follows:

[0061] Neighborhood extraction is performed for each pixel in the hyperspectral remote sensing image. The 11×11 range is extracted with the pixel to be classified as the center as the sample of the center pixel. The classification category of the sample adopts the real category corresponding to the center pixel. For pixels in the boundary area of ​​the hyperspectral remote sensing image, there will be insufficient neighboring pixels. This problem is solved by filling 5 rows and 5 columns of zero values ​​on the top, bottom, left and right of the hyperspectral image.

[0062] Step (3) divide the samples into training set and test set according to a certain ratio;

[0063] Furthermore, the specific implementation process of step (3) is as follows:

[0064] The sample data is divided into a training set and a test set at a ratio of 1:9. The division ratio can be adjusted in combination with the model training effect during implementation. The training set is used for model training, and the test set is used to test the model training effect.

[0065] Step (4) inputting the training set samples into the neural network in batches for training until the model is stable;

[0066] Furthermore, the specific implementation process of step (4) is as follows:

[0067] The small batch training method is used, and the batch size is set to 32, that is, 32 samples are taken from the training set each time and sent to the model for training. All samples in the training set are sent to the model for training once, which is called one round of training, and a total of 100 rounds of training are performed. The SGD optimizer is used for optimization, and the learning rate is set to 0.001, the weight decay is 0.0005, and the momentum is 0.9. The network is trained until the loss is close to 0 or stabilizes.

[0068] Step (5) uses the trained model to classify the hyperspectral image, evaluates the classification effect of the model through a pre-divided test set, and finally obtains the classification result graph.

[0069] Furthermore, in order to illustrate the effectiveness of the hyperspectral remote sensing image classification method of the present invention, the present invention is used to classify two public hyperspectral image datasets of Pavia University and Salinas. The conditions of the two datasets are shown in Table 1. The overall accuracy (OA), average accuracy (AA) and Kappa coefficient (Kappa) of different datasets are shown in Table 2. It can be seen that the present invention achieves 99.92% and 99.81% overall classification accuracy on the two public datasets of Pavia University and Salinas, respectively, when using 10% of the data as training samples. At the same time, the classification result diagrams of the two datasets are shown in Figures 1 and 2. Figure 4 , Figure 5 As shown, almost all pixels in the two data sets are classified correctly, and only very few pixels are misclassified into other categories. In addition, the present invention not only has high classification accuracy, but also obtains uniform and smooth classification result images with clear boundaries.

[0070] Table 1: Experimental dataset details

[0071]

[0072] Table 2: Classification accuracy of different datasets

[0073] Pavia University Salinas Overall accuracy (%) 99.92 99.81 Average accuracy (%) 99.89 99.88 Kappa coefficient 0.99 0.99

[0074] In summary, the present invention combines a 3D convolutional neural network, a 2D convolutional neural network, a multi-scale feature extraction and fusion module, and a spatial and spatial-spectral attention module to build a neural network for hyperspectral remote sensing image classification. On the basis of feature extraction by the convolutional neural network, a multi-scale feature extraction and fusion module based on a parallel convolutional block structure extracts multi-scale features in the feature map and fuses them. The spatial-spectral attention module and the spatial attention module calculate the correlation between pixels to further obtain a more complete and comprehensive feature map, thereby improving the classification accuracy of the network for hyperspectral remote sensing images.

[0075] Of course, the above examples of the present invention are only used to illustrate the implementation process of the present invention, and are not intended to limit the implementation methods of the present invention. A person of ordinary skill in the art may make corresponding changes and modifications according to the present invention, but these changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A hyperspectral remote sensing image classification model based on a hybrid convolutional neural network, characterized in that It includes multi-scale feature extraction and fusion module, spatial-spectral attention module, spatial attention module and classification module; The multi-scale feature extraction and fusion module includes a parallel convolution block structure, and the convolution layers of different branches in the module use convolution kernels of different sizes. A batch normalization layer and an activation layer are added after each convolution layer. The convolution operation is set to fill the feature map to keep the size of the feature map consistent. The feature maps I obtained from different branches are fused to obtain the feature map II that fuses the multi-scale features. The spatial-spectral attention module uses a non-local attention mechanism to perform attention operations on feature map III, and realizes data dimensionality reduction through a convolution layer with a convolution kernel size of 1, suppressing irrelevant information in feature map III; when calculating the output of each pixel position in the feature map, the non-local attention mechanism no longer only calculates the neighboring pixels but calculates the correlation of all position pixels in the feature map, and then uses the correlation as a weight to characterize the similarity between other position pixels and the current pixel to be calculated; since hyperspectral images have both spatial and spectral dimensions, non-local operations are performed in both spatial and spectral dimensions, and a more comprehensive feature map IV is obtained by calculating the correlation between pixels in the feature map; The spatial attention module also uses the non-local attention mechanism to perform attention operations on the feature map. It uses a 2D convolution kernel and performs non-local operations only in the spatial dimension to calculate the correlation of pixels at different spatial locations. The classification module includes a global average pooling layer, a fully connected layer, and a softmax classification function; the global average pooling layer aggregates the spatial feature maps extracted by the model, compresses the data volume, and then serves as the input of the fully connected layer, and finally uses the softmax function for classification; The feature map III is obtained through a 3D convolution layer and is used to be input into a spatial-spectral attention module; two layers of convolution layers + batch normalization layers and activation layers are connected in series after the spatial-spectral attention module to further extract the spatial-spectral feature map; two layers of 2D convolution layers + batch normalization layers and activation layers are added after the spatial attention module to further extract the spatial feature map; By using a 3D convolutional layer whose spectral dimension is equal to the number of spatial-spectral feature maps, its depth is normalized, and then the feature map V after removing the depth dimension is aggregated with the feature map II extracted by the multi-scale feature extraction module as the input of the spatial attention module; The implementation method of the classification model specifically includes the following steps: Step (1) connects the 3D convolutional neural network and the 2D convolutional neural network in series, adds parallel convolution blocks and non-local attention modules, and constructs a custom neural network; Step (2) divides the labeled hyperspectral image into several overlapping H×W×C small data cubes as the input of the model; Step (3) divide the samples into training set and test set according to a certain ratio; Step (4) inputting the training set samples into the neural network in batches for training until the model is stable; Step (5) classifies the hyperspectral image using the trained model, evaluates the classification effect of the model through a pre-divided test set, and finally obtains a classification result graph; The neural network in step (1) is divided into two branches, one of which is as follows: The first layer of the neural network in one branch is a 3D convolutional layer with a stride of 2, a convolution kernel size of 3×3×7, and a number of convolution kernels of 24. The information of the hyperspectral image is initially aggregated through the 3D convolutional layer to obtain feature map III, thereby reducing the amount of subsequent calculations. Next is a spatial-spectral attention module, which calculates the correlation of pixels in the spatial and spectral dimensions to obtain a more comprehensive feature map IV; Formula (1) is the general calculation formula of the non-local attention mechanism: Among them, y i Represents the output corresponding to the pixel at position i in the feature map, X i represents the pixel at position i in the input feature map, X j represents the pixels at all other positions in the feature map, c(X) is the normalization function; the f function in formula (1) is used to calculate the similarity between pixels, which is implemented by embedding Gaussian function, as shown in formula (2): Among them, θ(X i )=W θ X i , W θ and is the weight matrix to be learned, The g function is a linear transformation function, as shown in formula (3): g(X j )=W g X j (3) Among them, W g It is also the weight matrix to be learned. Normalization function c(X), as shown in formula (4): After the spatial-spectral attention module, there are two layers of 3D convolutional layers with a convolution kernel size of 3×3×5 and a number of 24, as well as corresponding batch normalization layers and activation layers, which are used to extract spatial-spectral information; followed by a layer of 3D convolutional layers with a convolution kernel size of 1×1×K and a number of 24 convolution kernels, as well as batch normalization layers and activation layers, which normalize the spectral dimension by using a convolution kernel scale with the same size as the spectral dimension of the spatial-spectral feature map in the spectral dimension to obtain the feature map V after compressing the spectral dimension; The other branch is a multi-scale feature extraction and fusion module. This module uses a parallel convolution block structure and multiple 2D convolution kernels of different sizes to extract multi-scale spatial features of hyperspectral images. It provides three sizes of receptive fields: 1×1, 3×3, and 5×5. It also uses 1×1 convolution and 3×3 maximum pooling to aggregate information. Finally, the feature maps I extracted by several convolution kernels are aggregated to obtain feature map II that fuses multi-scale features. The feature maps II and V obtained by the two branches of the neural network in step (1) are aggregated and then input into the spatial attention module to calculate the spatial correlation between pixels; The spatial features are further extracted through two layers of 2D convolutional layers with a convolution kernel size of 3×3 and a number of 24 + batch normalization layers and activation layers. Finally, the obtained feature maps are globally average pooled and connected to the fully connected layer for classification.

Images